From juan.campanario at UAH.ES Wed Mar 1 07:14:07 2006 From: juan.campanario at UAH.ES (Juan Miguel Campanario) Date: Wed, 1 Mar 2006 13:14:07 +0100 Subject: Errors in the Impact Factors computed and published by the ISI? Message-ID: Hello! This is my first message The impact factor of a given journal a given year (for example 2003) is computed as follows: Citations in 2003 to documents published in 2002 and 2001 IF (2003) = --------------------------------------------------------------------------------- Citable items published in 2002 and 2001 It seems to me that there are some errors in the numbers published by the ISI in the Web of Science. I have used the data stored in the Journal of Citation Report included in the version of Web of Science available for the universities in Spain. Table shows some errors that I have noticed. Please, could anyone tell me if I am doing something wrong or the errors are real? Journal ATHEROSCLEROSIS SUPPLEMENTS Year 2003 Citations 112 Items 35 IF (ISI) 4.457 Citations/Items=3.200 CA-A CANCER JOURNAL FOR CLINICIANS Year 2001 Citations 1078 Items 34 IF (ISI) 35.933 Citations/Items=31.706 Regards Juan Miguel Campanario (juan.campanario at uah.es) Departamento de F?sica Edificio de Ciencias, Universidad de Alcal?, 28871 Alcal? de Henares, Madrid (Spain) ------------------------------------------------------------------- Juan Miguel Campanario (http://www.uah.es/otrosweb/jmc) A solas mi vida paso, ni envidiado ni envidioso ------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From eugene.garfield at THOMSON.COM Wed Mar 1 14:01:34 2006 From: eugene.garfield at THOMSON.COM (Eugene Garfield) Date: Wed, 1 Mar 2006 14:01:34 -0500 Subject: FW: FINLAND SCIENCE AND TECHNOLOGY ASSESSMENT - 1 Message-ID: Go directly to: http://www.onr.navy.mil/sci_tech/special/354/technowatch/docs/FINLAND_DT IC_REPORT_8.pdf FROM: Dr. Ronald N. Kostoff (Office of Naval Research) TO: Finland Science and Technology Distribution SUBJECT: Finland S&T Assessment Report A report on the structure and infrastructure of Finnish science and technology is available for downloading (http://www.onr.navy.mil/sci_tech/special/354/technowatch/textmine.asp) . Go to eighth document listed. http://www.onr.navy.mil/sci_tech/special/354/technowatch/docs/FINLAND_DT IC_REPORT_8.pdf OVERVIEW A representative database of technical articles was extracted from the Science Citation Index for the years 2003-2004, with each article containing at least one author with a Finnish address. Document clustering was used to identify the main technical themes (core competencies) of Finnish research. Four of the pervasive technical topics identified from the clustering (Wireless Networks and Mobile Communication, Signal Processing, Materials Science and Engineering, Chemistry) were analyzed further using bibliometrics, in order to identify the infrastructure of these research areas. Finally, the citation performance of Finnish research in the four pervasive technical topics above, and in other technical topics obtained by text phrase analysis, was compared to that of two Scandanavian countries with similar populations and GDPs: Norway and Denmark. An Executive Summary of the approach and findings is appended. RNK REFERENCES 1. Kostoff RN, Tshiteya R., Bowles CA, Tuunanen T. The structure and infrastructure of the Finnish research literature. DTIC Technical Report ADA-442890 (http://www.dtic.mil/). Defense Technical Information Center. Fort Belvoir, VA. 2006. This report is based on the journal paper "Kostoff RN, Tshiteya R., Bowles CA, Tuunanen T. The structure and infrastructure of the Finnish research literature. Technology Analysis and Strategic Management. Vol. 18. 2006." APPENDIX - EXECUTIVE SUMMARY OF FINNISH S&T ASSESSMENT EXECUTIVE SUMMARY The main objective of this study was to assess the technical core competencies of Finland. This was accomplished using partitional clustering and bibliometrics on articles retrieved from the Science Citation Index. At the highest taxonomy level, there appear to be four major research core competencies, based on level of activity: * Physics/ Information/ Mathematics/ Economics includes about 28% of Finnish research; * Materials/Chemistry/ Environment/ Ecology includes about 28%; * Biology/ Biochemistry/ Medicine covers about 17%; and * Clinical Medicine includes about 27%. Bibliometrics were performed in detail on four lower level research core competencies identified by analysis of the overall technical taxonomy: * Wireless Network and Mobile Communications, * Signal Processing, * Materials Science and Engineering, and * Chemistry. Wireless Network and Mobile Communications * Journals containing the most papers appear to be applied journals, split between communications and networks. * Journals addressing more fundamental issues or topics from other disciplines are not represented. IEEE journals are represented most prominently. * Nokia and Helsinki University of Technology are by far the leaders for Wireless Network and Mobile Communications, with four institutions constituting the second tier: Oulu University, Tempere University of Technology, VTT, University of Helsinki. * There are nine universities, four research centers, and two industrial organizations. This distribution reflects an applied research/ technology development emphasis and is consistent with the journal section conclusions. * The major collaborators in this field are the advanced Western countries and Japan. Signal Processing * Journals containing the most papers are applied journals, drawing from a number of different applications areas. * Journals addressing more fundamental issues are not represented. * Helsinki University of Technology and Tampere University of Technology are the leading institutions for Signal Processing, with two institutions constituting the second tier: University of Oulu and University of Helsinki. * Four co-authoring countries (USA, England, France, Sweden) predominate in this field, accounting for 44% of the collaborative efforts. The main collaborators are the advanced Western countries. Materials Science and Engineering * Articles are published primarily in physics journals, although a small percentage of materials articles appear in materials and chemistry journals. * Journals addressing more fundamental issues, or topics from other disciplines, are not represented. * Helsinki University of Technology and University of Helsinki constitute the first tier for Materials Science and Engineering, with four universities and one research institute constituting the second tier (University Turku, VTT, Tampere University Technology, University Oulu, and Abo Akad University). * The USA is the predominant Materials research partner, with Germany, Sweden, and England constituting the second tier. Chemistry * The topical coverage of Chemistry journals is very broad, and the journals are a mix of basic (Journal of Biological Chemistry, Journal of Chemical Physics, Inorganic Chemistry) and applied (Applied Catalysis A, Nordic Pulp and Paper Research Journal, Industrial and Engineering Chemistry Research). * University of Helsinki is the leading Chemistry institution by far, with four institutions comprising the next tier (University Turku, Helsinki University of Technology, University Kuopio, and University Oulu), and two institutions that constitute the third major tier (Abo Akad University, Tampere University). * All of the institutions are universities. * The major collaborating country in Chemistry is the USA, with the next tier having three countries (Sweden, Germany, England). * The listing of top twenty countries aggregates various groupings (North American developed, Scandanavian countries, regional [Russia, Poland], developed Western democracies, and Finno-Ugric [Hungary, Estonia]). Integration of these bibliometrics results across the four technical thrusts studied leads to the following general conclusions: * The journals containing the most papers are mainly applied across the four core competencies selected, with some basic research journal representation in Chemistry. * University of Helsinki is in the top tier of prolific institutions in the more fundamental research areas, and is in the second tier in the more applied areas. * Helsinki University of Technology is in the top tier in more applied areas, and in the second tier in more fundamental areas. * University of Oulu is in the second tier in all four core technology areas selected. * Nokia is the only company to appear in a first tier (and only in the Wireless Network and Mobile Communications thrust). * VTT appears in a second tier (in the WNMC and MATLS thrusts). * The main country collaborators with Finland are the US (all four thrust areas), England (all four thrust areas), Germany (three thrust areas), Sweden (three thrust areas), France (two thrust areas). The clustering appears useful for generating the structure of a country's S&T, while the bibliometrics appears useful for identifying Centers of Excellence and prolific performers for specific technology areas. Continual upgrades in the clustering algorithms ensure that the accuracy of the clusters and categories will continue to improve. Two citation-based approaches were used for assessing Finland's technical thrust areas. One approach is based on themes from document clustering and compares the four themes used for the bibliometrics analysis. They are important research areas for Finland. The other approach is based on phrase frequency analysis, and identifies a number of pervasive technical themes that exhibit relatively high activity (numbers of articles published). For both approaches, the Finnish results are compared with those of two similar countries, Norway and Denmark. All three are small Scandanavian countries with similar populations and GDP. All three are viewed as advanced Western nations. Document Clustering-based Citations All the research articles on the four areas selected from the document clustering-based taxonomy published in 1999 with at least one author with a Finland or Norway or Denmark address were retrieved, and their citations examined. Then, global citation statistics were tabulated and compared for each country for each technology. Wireless Networks and Mobile Communications The number of Finnish articles in Wireless Networks and Mobile Communications published in both 1999 and 2004 is almost an order of magnitude larger than that from Norway or Denmark. This investment reflects an obvious strategic priority for Finland, and is one characteristic of a true core competency. Because of the small number of data points for Norway and Denmark for the citation comparison vintage year (1999), the citation comparison statistics for Wireless Networks and Mobile Communications have limited validity. For example, the median of the top twenty percent of articles for Norway and Denmark reflected one article for each country. In addition, many of the Norway and Denmark articles did not focus on development of the wireless/ mobile technology but rather on consequences such as health effects of microwave radiation. Finland did have three highly cited papers (>40), but the relative significance of this number could not be ascertained due to the poor statistics for the other two countries. To place this result in a larger context, the citation performance of the USA in Wireless Networks and Mobile Communications was examined for articles published in 1999 and compared to that of Finland. * There were 943 USA articles published in 1999 (compared to Finland's 106). * The median of the top twenty percent USA article citations was 32 (compared to Finland's 35). * The median of all the USA article citations was 5 (compared to Finland's 2). * Finally, the number of USA articles with forty or more citations was 69 (compared to Finland's 3). Thus, in relative terms, Finland's research output in Wireless Networks and Mobile Communications reflects a higher priority than the USA's, but in absolute terms, the USA is dominant by a large margin. Signal Processing In the Signal Processing category, Norway has increased its output substantially since 1999, while Finland's output has increased moderately, and Denmark's has remained the same. Denmark exceeds Finland and Norway noticeably in all the citation metrics used. Most importantly, Denmark has a higher fraction of the most highly cited articles. Thus, while Signal Processing may be a core Finnish research area due to volume of activity, it does not offer a core research impact advantage over Denmark. Materials In the Materials category, Finland and Denmark are the clear leaders over Norway in numbers of research articles published, with Finland holding a slight edge. Both Finland and Denmark have increased publication productivity by about a third since 1999, while Norway's increase has been about half that amount. As in the Signal Processing thrust, Denmark outperforms Finland in all the citation metrics used, most importantly in the number of highly cited papers. Again, as in the Signal Processing thrust, Materials research may be a core Finnish research area due to volume of activity, but it certainly does not offer a core research impact advantage over Denmark. Chemistry In the Chemistry category, Finland and Denmark are the clear leaders in numbers of publications and are relatively close to each other. Again, Denmark is the clear leader in all the citation metrics used, with moderately less of the commanding lead shown in Materials research. Three caveats are in order here. The first concerns the relation between citations, research impact, and core competencies. Poor citation performance can reflect: * Poor intrinsic quality, and/ or * Low circulation journals, and/ or * Low research activity in field, and or * More applied focus, reducing the number of extra-discipline researchers available to cite, and/ or * Other characteristics. In particular, a country could produce high quality but very applied research in a technical area. The area's output might receive low citations in aggregate but still be a national core competence. In order to distinguish the reasons for poor citations exhibited by different countries in different categories, one would need to read substantial numbers of research articles produced by the countries in the field of interest and analyze them for intrinsic quality and level of development. That was beyond the scope of the present study. The second caveat concerns the aggregation level of the technologies examined. The conclusions above about dominance are for the thrust areas defined by the relatively broad queries shown. Even though one country may dominate on the thrust area in aggregate (as defined by the total query), another country could conceiveably lead in one of the sub-thrust areas. The third caveat concerns the selection of phrases. The selection was based on emphasis in the Finland research output database. A priori, one would expect these to be Finnish research priority areas and would expect Finland's performance relative to other countries to be somewhat higher than average. Any poor relative performance by Finland in these Finnish thrust areas should be cause for concern. In the above four thrust areas analyzed, Finland has two characteristics of a true core competency: an identifiable cohesive thrust and identifiable critical mass (numbers of publications, relative to Norway and Denmark). However, for the latter three technical thrust areas, it did not have the aggregate citation impact of Denmark. The differences in relative publication and citation attributes across the four technical areas for the three countries show the necessity for comparing countries at the critical technology level rather than at the aggregate national level (King, 2004). Phrase Frequency-based Citations A phrase frequency analysis was performed on the contents of the Abstracts of all the Finland records retrieved. The highest frequency technical phrases were identified. Seventeen of these phrases were selected from the areas of Physical, Environmental, and Life Sciences, with emphasis on phrases different from those used for the core competency queries. No high frequency phrases of adequate specificity could be identified from the Engineering Sciences area. Each phrase (in some cases, combinations of similar phrases) was entered into the SCI search engine, and records were retrieved for Finland, Norway, and Denmark for 1999. Citation indicators were applied to each phrase's retrievals, and the results compared. The phrase frequency approach shows: * Finland to be the clear winner in most of the citation metrics in most of the Life Sciences themes, * Denmark to be the winner in most Environmental Sciences themes, * Denmark to be a clear winner in four of the seven Physical Sciences themes, * Finland to be a modest winner in one Physical Sciences theme (Electron Microscopy), and * Finland and Denmark to be essentially tied in three Physical Sciences themes. The caveats expressed previously about the interpretation of citations and thematic aggregation apply to these results as well. The real surprise in these results is the citation performance of Norway. While Norway has the smallest population of the three countries, it has the largest GDP. In most cases (but not all), Norway had the lowest research activity, and in almost all cases, had the lowest citation impact. It should be re-stated that the phrases selected for the present country comparison analysis, whether from the document clustering-based queries or the phrase frequency-based queries, were based on emphasis in the Finland data base. Whether Norway would perform better if phrases were selected from an analysis of its own database is unknown. As will be shown in the next paragraph, its gross citation numbers do not offer much encouragement, but as this study has emphasized repeatedly, the dis-aggregated critical technologies are of most interest. Computations were performed to identify the effect of collaborating countries on the research impact of articles with Finnish authors. There are four separate groups of results generated. The first group, Total Country Results, is based on all articles from either 2004 or 1999 that had at least one author with the address of the country of interest (i.e., Finland or Norway or Denmark). The second group, Country Only (No Other Countries), is based on all articles from 1999 with at least one author with the address of the country of interest, excluding all authors with an address from any of Finland's 25 top country collaborators. The purpose of this group is to identify citation characteristics of papers from only the country of interest (Finalnd, Norway, or Denmark). The third group, Country and USA Only (No Other Countries) is based on all articles from 1999 with at least one author with the address of the country of interest and at least one author with a USA address, excluding all other authors with an address from any of Finland's 25 top country collaborators. The purpose of this group is to identify the impact of adding only USA co-authors to those of the country of interest. The final group, Country and USA, is based on all articles from 1999 with at least one author from the USA and one author from the country of interest. Papers with authors from other countries are not excluded, as they were in the third group. The summary findings are as follows: * In the first group, Total Country Results, the gross citation impacts of Finland and Denmark are quite similar, and substantially larger than that of Norway. * In the second group, Country Only, Denmark has modestly better citation performance than Finland, which in turn has modestly better citation performance than Norway. * In the third group, Country and USA Only (No Other Countries), Finland and the USA outperform Denmark and the USA. Both groups substantially outperform Norway and the USA. * In the fourth group, Country and USA, multi-country papers including Finland and USA very slightly outperform multi-country papers including Denmark and USA, which in turn moderately outperform multi-country papers including Norway and the USA. * Adding the USA only to Finland/ Norway/ Denmark papers increases the citation performance substantially, and adding more countries enhances performance modestly for Finland, somewhat more for Denmark, and substantially for Norway. The citation-based approaches appear very useful for comparing research impact among countries, but substantial reading of research outputs is required for proper interpretation of citation results. -------------- next part -------------- An HTML attachment was scrubbed... URL: From juan.campanario at UAH.ES Thu Mar 2 05:44:41 2006 From: juan.campanario at UAH.ES (Juan Miguel Campanario) Date: Thu, 2 Mar 2006 11:44:41 +0100 Subject: Problems with impact factors computed and published by the ISI? Message-ID: Dear people: Many thanks for your e-mail messages concerning my post on problems with impact factors. I did not computed citations. I just used the numbers publised by the ISI in the JCR. Below is an excerpt (copy+paste) I took from the JCR version available to universities in Spain. I just reproduced the computations. I would be glad to hear other people who has access to the JCR. Please,could you tell me if I am using the right numbers? Regards Juan Miguel Campanario http://www.uah.es/otrosweb/jmc ATHEROSCLEROSIS SUPPLEMENTS 2003 IMPACT FACTOR 342e13.jpg Journal Impact Factor 342e31.jpg Cites in 2003 to articles published in: 2002 = 63 Number of articles published in: 2002 = 24 2001 = 49 2001 = 11 Sum: 112 Sum: 35 Calculation: Cites to recent articles 112 = 4.457 Number of recent articles 35 Journal Impact Factor 342e63.jpg ERROR: 112 / 35 = 3.3 CA-A CANCER JOURNAL FOR CLINICIANS 2001 IMPACT FACTOR Cites in 2001 to articles published in: 2000 = 595 Number of articles published in: 2000 = 16 1999 = 483 1999 = 18 Sum: 1078 Sum: 34 Calculation: Cites to recent articles 1078 = 35.933 Number of recent articles 34 ERROR: 1078 / 34 = 31.706 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 342e13.jpg Type: image/jpeg Size: 1083 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 342e31.jpg Type: image/jpeg Size: 980 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 342e63.jpg Type: image/jpeg Size: 980 bytes Desc: not available URL: -------------- next part -------------- ------------------------------------------------------------------- Juan Miguel Campanario (http://www.uah.es/otrosweb/jmc) A solas mi vida paso, ni envidiado ni envidioso ------------------------------------------------------------------- From juan.campanario at UAH.ES Thu Mar 2 05:45:53 2006 From: juan.campanario at UAH.ES (Juan Miguel Campanario) Date: Thu, 2 Mar 2006 11:45:53 +0100 Subject: Problems with impact factors computed and published by the ISI?-2 Message-ID: Dear people: Many thanks for your e-mail messages concerning my post on problems with impact factors. I did not computed citations. I just used the numbers publised by the ISI in the JCR. Below is an excerpt (copy+paste) I took from the JCR version available to universities in Spain. I just reproduced the computations. I would be glad to hear other people who has access to the JCR. Please,could you tell me if I am using the right numbers? Regards Juan Miguel Campanario http://www.uah.es/otrosweb/jmc ATHEROSCLEROSIS SUPPLEMENTS 2003 IMPACT FACTOR 354a27.jpg Journal Impact Factor 354a31.jpg Cites in 2003 to articles published in: 2002 = 63 Number of articles published in: 2002 = 24 2001 = 49 2001 = 11 Sum: 112 Sum: 35 Calculation: Cites to recent articles 112 = 4.457 Number of recent articles 35 Journal Impact Factor 354a3b.jpg ERROR: 112 / 35 = 3.2 CA-A CANCER JOURNAL FOR CLINICIANS 2001 IMPACT FACTOR Cites in 2001 to articles published in: 2000 = 595 Number of articles published in: 2000 = 16 1999 = 483 1999 = 18 Sum: 1078 Sum: 34 Calculation: Cites to recent articles 1078 = 35.933 Number of recent articles 34 ERROR: 1078 / 34 = 31.706 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 354a27.jpg Type: image/jpeg Size: 1083 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 354a31.jpg Type: image/jpeg Size: 980 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 354a3b.jpg Type: image/jpeg Size: 980 bytes Desc: not available URL: -------------- next part -------------- ------------------------------------------------------------------- Juan Miguel Campanario (http://www.uah.es/otrosweb/jmc) A solas mi vida paso, ni envidiado ni envidioso ------------------------------------------------------------------- From garfield at CODEX.CIS.UPENN.EDU Thu Mar 2 15:40:17 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Thu, 2 Mar 2006 15:40:17 -0500 Subject: Ball P. "Prestige is factored into journal ratings " NATURE 439 (7078): 770-771 FEB 16 2006 Message-ID: FULL TEXT AVAILABLE AT : http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR http://guide.labanimal.com/news/2006/060213/full/439770a.html Philip Ball : p.ball at nature.com www.philipball.com Title: Prestige is factored into journal ratings Author(s): Ball P Source: NATURE 439 (7078): 770-771 FEB 16 2006 Document Type: News Item Language: English Cited References: 0 Times Cited: 0 Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 CRINAN ST, LONDON N1 9XW, ENGLAND Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA ISSN: 0028-0836 From garfield at CODEX.CIS.UPENN.EDU Fri Mar 3 12:37:07 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Fri, 3 Mar 2006 12:37:07 -0500 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: Further to yesterday's posting, "Prestige is factored into journal ratings", here is another interesting and informative article FULL TEXT AVAILABLE AT : http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf email: {jbollen, marko, herbertv}@lanl.gov TITLE : Journal Status AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 Abstract The status of an actor in a social context is commonly defined in terms of two factors: the total number of endorsements the actor receives from other actors and the prestige of the endorsing actors. These two factors indicate the distinction between popularity and expert appreciation of the actor, respectively. We refer to the former as popularity and to the latter as prestige. These notions of popularity and prestige also apply to the domain of scholarly assessment. The ISI Impact Factor (ISI IF) is defined as the mean number of citations a journal receives over a 2 year period. By merely counting the amount of citations and disregarding the prestige of the citing journals, the ISI IF is a metric of popularity, not of prestige. We demonstrate how a weighted version of the popular PageRank algorithm can be used to obtain a metric that reflects prestige. We contrast the rankings of journals according to their ISI IF and their weighted PageRank, and we provide an analysis that reveals both significant overlaps and differences. Furthermore, we introduce the Y-factor which is a simple combination of both the ISI IF and the weighted PageRank, and find that the resulting journal rankings correspond well to a general understanding of journal status. ______________________________________________ FULL TEXT AVAILABLE AT : http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR http://guide.labanimal.com/news/2006/060213/full/439770a.html Philip Ball : p.ball at nature.com www.philipball.com Title: Prestige is factored into journal ratings Author(s): Ball P Source: NATURE 439 (7078): 770-771 FEB 16 2006 Document Type: News Item Language: English Cited References: 0 Times Cited: 0 Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 CRINAN ST, LONDON N1 9XW, ENGLAND Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA ISSN: 0028-0836 From loet at LEYDESDORFF.NET Sat Mar 4 08:14:57 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Sat, 4 Mar 2006 14:14:57 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: Dear colleagues, The idea is interesting. However, there a few problems with this paper. First, the authors should not have used Pearson correlation coefficients to compare the rankings, but rank correlations (Spearman's rho or Kendall's tau). Second, it would have been interesting to have a rank correlation with "total cites" given recent discussions (Bensman). Third, the delineation of fields in terms of the ISI subject categories is very questionnable. However, the authors are very clear about their results: "We identified ... , but were unable to recognize a meaningful pattern in the results." (p. 9). I don't understand why one should then multiply the one measure with the other. What does multiplication to the error? Does one of you know a place where the ISI subject categories are justified? How are they produced? People seem to use them increasingly both in evaluation and research practices, but I have never been able to reproduce them using journal citation measures. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Eugene Garfield > Sent: Friday, March 03, 2006 6:37 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and > Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Further to yesterday's posting, "Prestige is factored into > journal ratings", here is another interesting and informative article > > FULL TEXT AVAILABLE AT : > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf > > email: {jbollen, marko, herbertv}@lanl.gov > > TITLE : Journal Status > > AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel > > SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Abstract > The status of an actor in a social context is commonly > defined in terms of two factors: the total number of > endorsements the actor receives from other actors and the > prestige of the endorsing actors. These two factors indicate > the distinction between popularity and expert appreciation of > the actor, respectively. We refer to the former as popularity > and to the latter as prestige. These notions of popularity > and prestige also apply to the domain of scholarly > assessment. The ISI Impact Factor (ISI IF) is defined as the > mean number of citations a journal receives over a 2 year > period. By merely counting the amount of citations and > disregarding the prestige of the citing journals, the ISI IF > is a metric of popularity, not of prestige. We demonstrate > how a weighted version of the popular PageRank algorithm can > be used to obtain a metric that reflects prestige. We > contrast the rankings of journals according to their ISI IF > and their weighted PageRank, and we provide an analysis that > reveals both significant overlaps and differences. > Furthermore, we introduce the Y-factor which is a simple > combination of both the ISI IF and the weighted PageRank, and > find that the resulting journal rankings correspond well to a > general understanding of journal status. > > > ______________________________________________ > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > FULL TEXT AVAILABLE AT : > http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR > http://guide.labanimal.com/news/2006/060213/full/439770a.html > > > Philip Ball : p.ball at nature.com > www.philipball.com > > Title: Prestige is factored into journal ratings > > Author(s): Ball P > > Source: NATURE 439 (7078): 770-771 FEB 16 2006 > > Document Type: News Item Language: English > Cited References: 0 Times Cited: 0 > > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 > CRINAN ST, LONDON > N1 9XW, ENGLAND > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA > > ISSN: 0028-0836 > From juan.campanario at UAH.ES Mon Mar 6 05:51:53 2006 From: juan.campanario at UAH.ES (Juan Miguel Campanario) Date: Mon, 6 Mar 2006 11:51:53 +0100 Subject: Another error in Impact Factors? Message-ID: Hello: Here is another problem with the calculation of IF. I just copied from the ISI Web of Science (JCR) EE-EVALUATION ENGINEERING Year=2004 Journal Impact Factor 97c37d.jpg Cites in 2004 to articles published in: 2003 = 0 Number of articles published in: 2003 = 88 2002 = 0 2002 = 85 Sum: 0 Sum: 173 Calculation: Cites to recent articles 0 = 0.006 Number of recent articles 173 Maybe I am using the wrong numbers, but I just copied from the JCR. What happens here? Regards Juan Miguel Campanario -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 97c37d.jpg Type: image/jpeg Size: 980 bytes Desc: not available URL: -------------- next part -------------- ------------------------------------------------------------------- Juan Miguel Campanario (http://www.uah.es/otrosweb/jmc) A solas mi vida paso, ni envidiado ni envidioso ------------------------------------------------------------------- From loet at LEYDESDORFF.NET Mon Mar 6 06:33:17 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Mon, 6 Mar 2006 12:33:17 +0100 Subject: Another error in Impact Factors? In-Reply-To: <6.0.0.22.1.20060306112555.0244ef80@mail.uah.es> Message-ID: Dear Juan, The IF = 0.006 points to a 1 citation. 1/173 = 0.00578. Single citations are lumped together under "all others" by the ISI? However, I did not know that this was done also for the individual years. Best, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ _____ From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Juan Miguel Campanario Sent: Monday, March 06, 2006 11:52 AM To: SIGMETRICS at LISTSERV.UTK.EDU Subject: [SIGMETRICS] Another error in Impact Factors? Hello: Here is another problem with the calculation of IF. I just copied from the ISI Web of Science (JCR) EE-EVALUATION ENGINEERING Year=2004 Journal Impact Factor 97c37d.jpg Cites in 2004 to articles published in: 2003 = 0 Number of articles published in: 2003 = 88 2002 = 0 2002 = 85 Sum: 0 Sum: 173 Calculation: Cites to recent articles 0 = 0.006 Number of recent articles 173 Maybe I am using the wrong numbers, but I just copied from the JCR. What happens here? Regards Juan Miguel Campanario -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 97c37d.jpg Type: image/jpeg Size: 980 bytes Desc: not available URL: From juan.campanario at UAH.ES Mon Mar 6 06:34:48 2006 From: juan.campanario at UAH.ES (Juan Miguel Campanario) Date: Mon, 6 Mar 2006 12:34:48 +0100 Subject: Problems with the 2003 Impact Factor of Energy Exploration and Exploitation? Message-ID: I am studying now the 2003 Impact Factor of ENERGY EXPLORATION & EXPLOITATION According the ISI, there are 5 citations that contribute to the Impact Factor Cites in 2003 to articles published in: 2002 = 3 2001 = 2 SUM: 5 Number of articles published in: 2002 = 35 2001 = 25 Sum: 60 Calculation: Cites to recent articles 5 = 0.083 Number of recent articles 60 Now, when we go to the "Cited Journal" and we download the "Cited Journal data table" it seems to me that there are some problems. Citations from ALL JOURNALS, 2002= 3 Citatons from ALL JOURNALS, 2001= 2 I guess that these are the 5 citations that contribute to the Impact Factor. However, the column 2002 only shows 1 citation (from Energy Explot Exploit) (see the atached file) The column 2001 shows 0 citations So, it seems that there are 4 missing citations The sum of citations in other columns (2000, 1999, ...) seems to be O.K. Now, if we add the numbers if row "All Journals", the total number is 27, not 31 as stated in the table. Now, if we add the numbers corresponding to journals and years, we obtain 23, for rows and columns (yelow and orange, in the atached file). It seems again that there are 4 citations that have dissapeared.... Note that I am using the data from the ISI, I am not computing citations. Is there something wrong with my data or computations? Regards Juan Miguel Campanario -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: energytabla.xls Type: application/octet-stream Size: 15872 bytes Desc: not available URL: -------------- next part -------------- ------------------------------------------------------------------- Juan Miguel Campanario (http://www.uah.es/otrosweb/jmc) A solas mi vida paso, ni envidiado ni envidioso ------------------------------------------------------------------- From juan.campanario at UAH.ES Mon Mar 6 06:42:52 2006 From: juan.campanario at UAH.ES (Juan Miguel Campanario) Date: Mon, 6 Mar 2006 12:42:52 +0100 Subject: Impact factor of EE-EVALUATION ENGINEERING Message-ID: Dear Loet: I have the data in my screen: Cites in 2004 to articles published in: 2003 = 0 2002=0 Sum=0 Number of articles published in: 2003 = 88 2002 = 85 Sum: 173 Calculation: Cites to recent articles 0 = 0.006 Number of recent articles 173 I have checked also the cited journal data table. It seems that there are 0 citations (0+0 = 0). Please, could you check your local version of JCR? This is amazing.... Juan Miguel Campanario ------------------------------------------------------------------- Juan Miguel Campanario (http://www.uah.es/otrosweb/jmc) A solas mi vida paso, ni envidiado ni envidioso ------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From juan.campanario at UAH.ES Mon Mar 6 07:11:17 2006 From: juan.campanario at UAH.ES (Juan Miguel Campanario) Date: Mon, 6 Mar 2006 13:11:17 +0100 Subject: More missing citations? Message-ID: Hello again More missing citations? I have downloaded now the CITED JOURNAL DATA TABLE corresponding to the journal IEEE TRANSACTIONS ON COMPONENTS AND PACKAGING TECHNOLOGIES, Year=2003 According to the ISI the computation is as follows Cites in 2003 to articles published in: 2002 = 43 2001= 107 Sum: 150 Number of articles published in: 2002 = 94 2001 = 96 Sum: 190 Calculation: Cites to recent articles 150 = 0.790 Number of recent articles 190 Now, if we go to the ?cited journal data table?, it seems that the number of citations is different (attached file, in ORANGE) I obtain 17 citations in 2002 and 18 citations in 2001 (35 citations that contribute to the Impact Factor) I have checked the data, but I cannot find the error Regards Juan Miguel Campanario -------------- next part -------------- A non-text attachment was scrubbed... Name: ieee.xls Type: application/octet-stream Size: 38400 bytes Desc: not available URL: -------------- next part -------------- ------------------------------------------------------------------- Juan Miguel Campanario (http://www.uah.es/otrosweb/jmc) A solas mi vida paso, ni envidiado ni envidioso ------------------------------------------------------------------- From loet at LEYDESDORFF.NET Mon Mar 6 07:32:08 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Mon, 6 Mar 2006 13:32:08 +0100 Subject: Impact factor of EE-EVALUATION ENGINEERING In-Reply-To: <6.0.0.22.1.20060306123927.0265ad60@mail.uah.es> Message-ID: Yes, I have similar values. Best, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ _____ From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Juan Miguel Campanario Sent: Monday, March 06, 2006 12:43 PM To: SIGMETRICS at LISTSERV.UTK.EDU Subject: [SIGMETRICS] Impact factor of EE-EVALUATION ENGINEERING I have the data in my screen: Cites in 2004 to articles published in: 2003 = 0 2002=0 Sum=0 Number of articles published in: 2003 = 88 2002 = 85 Sum: 173 Calculation: Cites to recent articles 0 = 0.006 Number of recent articles 173 I have checked also the cited journal data table. It seems that there are 0 citations (0+0 = 0). Please, could you check your local version of JCR? This is amazing.... Juan Miguel Campanario ------------------------------------------------------------------- Juan Miguel Campanario (http://www.uah.es/otrosweb/jmc) A solas mi vida paso, ni envidiado ni envidioso ------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From juan.campanario at UAH.ES Mon Mar 6 07:39:43 2006 From: juan.campanario at UAH.ES (Juan Miguel Campanario) Date: Mon, 6 Mar 2006 13:39:43 +0100 Subject: Impact factor of EE-EVALUATION ENGINEERING In-Reply-To: Message-ID: Dear Loet: Many thanks. It seems that we have a very simple problem with citation counts I have asked to the ISI Technical support. Juan Miguel Campanario At 13:32 06/03/2006, you wrote: >Adminstrative info for SIGMETRICS (for example unsubscribe): >http://web.utk.edu/~gwhitney/sigmetrics.html >Yes, I have similar values. > >Best, Loet > >________________________________ >Loet Leydesdorff >Amsterdam School of Communications Research (ASCoR), >Kloveniersburgwal 48, 1012 CX Amsterdam. >Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; >loet at leydesdorff.net ; >http://www.leydesdorff.net/ > > > > >---------- >From: ASIS&T Special Interest Group on Metrics >[mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Juan Miguel Campanario >Sent: Monday, March 06, 2006 12:43 PM >To: SIGMETRICS at LISTSERV.UTK.EDU >Subject: [SIGMETRICS] Impact factor of EE-EVALUATION ENGINEERING > >Adminstrative info for SIGMETRICS (for example unsubscribe): >http://web.utk.edu/~gwhitney/sigmetrics.html Dear Loet: > >I have the data in my screen: > >Cites in 2004 to articles published in: 2003 = 0 >2002=0 >Sum=0 > > >Number of articles published in: 2003 = 88 >2002 = 85 >Sum: 173 > >Calculation: Cites to recent articles 0 = 0.006 > Number of recent articles 173 > >I have checked also the cited journal data table. >It seems that there are 0 citations (0+0 = 0). >Please, could you check your local version of JCR? > >This is amazing.... > >Juan Miguel Campanario >------------------------------------------------------------------- >Juan Miguel Campanario (http://www.uah.es/otrosweb/jmc) > >A solas mi vida paso, ni envidiado ni envidioso >------------------------------------------------------------------- ------------------------------------------------------------------- Juan Miguel Campanario (http://www.uah.es/otrosweb/jmc) A solas mi vida paso, ni envidiado ni envidioso ------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From meho at INDIANA.EDU Mon Mar 6 09:14:50 2006 From: meho at INDIANA.EDU (Lokman I. Meho) Date: Mon, 6 Mar 2006 09:14:50 -0500 Subject: More missing citations? In-Reply-To: <6.0.0.22.1.20060306130227.02635b30@mail.uah.es> Message-ID: Please note that the JIF formula for 2003 impact factor is: Number of Citations Received in 2003 to Citable Items Published in 2001-2002 / Number of Citable Items Published in 2001-2002 According to ISI, "Citable Items" are Journal and Review Articles. This means you have to limit your searches/counts to these two document types only. As for the journal in question, note that the journal changed its name to IEEE Transactions on Advanced Packaging. When you take this into consideration, the number of times the 2001-2002 items (i.e., 190 journal and review articles) were cited is in fact 150. 150/190 = .790 Best, Lokman Lokman I. Meho, Ph.D. Assistant Professor School of Library and Information Science Indiana University 1320 East 10th Street, LI 011 Bloomington, IN 47405-3907 Tel: (812) 856-2323 Fax: (812) 855-6166 E-mail: meho at indiana.edu http://www.slis.indiana.edu/faculty/meho/ Quoting Juan Miguel Campanario : > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Hello again > > More missing citations? > > I have downloaded now the CITED JOURNAL DATA TABLE corresponding to > the journal > IEEE TRANSACTIONS ON COMPONENTS AND PACKAGING TECHNOLOGIES, > Year=2003 > > According to the ISI the computation is as follows > Cites in 2003 to articles published in: 2002 = 43 > 2001= 107 > Sum: 150 > > Number of articles published in: 2002 = 94 > 2001 = 96 > Sum: 190 > > Calculation: Cites to recent articles 150 = 0.790 > Number of recent articles 190 > > Now, if we go to the ?cited journal data table?, it seems that the > number of citations is different (attached file, in ORANGE) > > I obtain 17 citations in 2002 and 18 citations in 2001 (35 citations > that contribute to the Impact Factor) > > I have checked the data, but I cannot find the error > > Regards > > Juan Miguel Campanario From davisc at INDIANA.EDU Mon Mar 6 09:53:06 2006 From: davisc at INDIANA.EDU (Charles H. Davis) Date: Mon, 6 Mar 2006 09:53:06 -0500 Subject: More missing citations? In-Reply-To: <20060306091450.s88y30i99ccc8s4k@webmail-beta.iu.edu> Message-ID: Nice catch, Lokman! :-) Chuck Quoting "Lokman I. Meho" : > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Please note that the JIF formula for 2003 impact factor is: > > Number of Citations Received in 2003 to Citable Items Published in > 2001-2002 / Number of Citable Items Published in 2001-2002 > > According to ISI, "Citable Items" are Journal and Review Articles. This > means you have to limit your searches/counts to these two document > types only. > > As for the journal in question, note that the journal changed its name > to IEEE Transactions on Advanced Packaging. When you take this into > consideration, the number of times the 2001-2002 items (i.e., 190 > journal and review articles) were cited is in fact 150. > > 150/190 = .790 > > Best, > Lokman > > Lokman I. Meho, Ph.D. > Assistant Professor > School of Library and Information Science > Indiana University > 1320 East 10th Street, LI 011 > Bloomington, IN 47405-3907 > Tel: (812) 856-2323 > Fax: (812) 855-6166 > E-mail: meho at indiana.edu > http://www.slis.indiana.edu/faculty/meho/ > > > > > Quoting Juan Miguel Campanario : > >> Adminstrative info for SIGMETRICS (for example unsubscribe): >> http://web.utk.edu/~gwhitney/sigmetrics.html >> >> Hello again >> >> More missing citations? >> >> I have downloaded now the CITED JOURNAL DATA TABLE corresponding to >> the journal >> IEEE TRANSACTIONS ON COMPONENTS AND PACKAGING TECHNOLOGIES, >> Year=2003 >> >> According to the ISI the computation is as follows >> Cites in 2003 to articles published in: 2002 = 43 >> 2001= 107 >> Sum: 150 >> >> Number of articles published in: 2002 = 94 >> 2001 = 96 >> Sum: 190 >> >> Calculation: Cites to recent articles 150 = 0.790 >> Number of recent articles 190 >> >> Now, if we go to the ?cited journal data table?, it seems that the >> number of citations is different (attached file, in ORANGE) >> >> I obtain 17 citations in 2002 and 18 citations in 2001 (35 citations >> that contribute to the Impact Factor) >> >> I have checked the data, but I cannot find the error >> >> Regards >> >> Juan Miguel Campanario > From tdb01r at ECS.SOTON.AC.UK Mon Mar 6 10:28:33 2006 From: tdb01r at ECS.SOTON.AC.UK (Tim Brody) Date: Mon, 6 Mar 2006 15:28:33 +0000 Subject: More missing citations? In-Reply-To: <20060306095306.0bq4i75buow4o40g@webmail-beta.iu.edu> Message-ID: Hmmm (http://en.wikipedia.org/wiki/Impact_Factor), A = the number of times articles published in 2001-2 were cited in tracked journals during 2003 B = the number of articles published in 2001-2 2003 impact factor = A/B "ISI excludes certain article types (such as news items, correspondence, and errata) from the *denominator*" (my emphasis) Without trawling for an 'official' ISI definition that is what I understood the IF to be. One of the oft-cited criticisms of the JIF being that journals with many letters that get cited get an unfair advantage due to letters being included in the numerator but not the denominator. Is this anomaly corrected in recent JCRs (in which case wikipedia at least needs to be updated)? How does that effect time-series comparisons? Thanks, Tim. Charles H. Davis wrote: > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Nice catch, Lokman! :-) > > Chuck > > Quoting "Lokman I. Meho" : > >> Adminstrative info for SIGMETRICS (for example unsubscribe): >> http://web.utk.edu/~gwhitney/sigmetrics.html >> >> Please note that the JIF formula for 2003 impact factor is: >> >> Number of Citations Received in 2003 to Citable Items Published in >> 2001-2002 / Number of Citable Items Published in 2001-2002 >> >> According to ISI, "Citable Items" are Journal and Review Articles. This >> means you have to limit your searches/counts to these two document >> types only. >> >> As for the journal in question, note that the journal changed its name >> to IEEE Transactions on Advanced Packaging. When you take this into >> consideration, the number of times the 2001-2002 items (i.e., 190 >> journal and review articles) were cited is in fact 150. >> >> 150/190 = .790 >> >> Best, >> Lokman >> >> Lokman I. Meho, Ph.D. >> Assistant Professor >> School of Library and Information Science >> Indiana University >> 1320 East 10th Street, LI 011 >> Bloomington, IN 47405-3907 >> Tel: (812) 856-2323 >> Fax: (812) 855-6166 >> E-mail: meho at indiana.edu >> http://www.slis.indiana.edu/faculty/meho/ >> >> >> >> >> Quoting Juan Miguel Campanario : >> >>> Adminstrative info for SIGMETRICS (for example unsubscribe): >>> http://web.utk.edu/~gwhitney/sigmetrics.html >>> >>> Hello again >>> >>> More missing citations? >>> >>> I have downloaded now the CITED JOURNAL DATA TABLE corresponding to >>> the journal >>> IEEE TRANSACTIONS ON COMPONENTS AND PACKAGING TECHNOLOGIES, >>> Year=2003 >>> >>> According to the ISI the computation is as follows >>> Cites in 2003 to articles published in: 2002 = 43 >>> 2001= 107 >>> Sum: 150 >>> >>> Number of articles published in: 2002 = 94 >>> 2001 = 96 >>> Sum: 190 >>> >>> Calculation: Cites to recent articles 150 = 0.790 >>> Number of recent articles 190 >>> >>> Now, if we go to the ?cited journal data table?, it seems that the >>> number of citations is different (attached file, in ORANGE) >>> >>> I obtain 17 citations in 2002 and 18 citations in 2001 (35 citations >>> that contribute to the Impact Factor) >>> >>> I have checked the data, but I cannot find the error? >>> >>> Regards >>> >>> Juan Miguel Campanario >> >> > From kboyack at SANDIA.GOV Mon Mar 6 10:42:29 2006 From: kboyack at SANDIA.GOV (Boyack, Kevin W) Date: Mon, 6 Mar 2006 08:42:29 -0700 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: Hello everyone, Regarding ISI subject categories, I don't know how they are picked, but you can read about journal inclusion at http://scientific.thomson.com/free/essays/selectionofmaterial/journalsel ection/. I would assume that if this much effort is going into picking journals to index, there is at least some level of thought going into subject categorization. After all, the things ISI is looking at to decide on journal inclusion are in many cases the same things you would look at for categorization. Now I won't claim that ISI's categorization is perfect, or that it is updated as frequently as it might be, but I have yet to see anyone put forth a better categorization scheme that is widely available. I'm sure there are many of us that either have, or could easily, make up our own schemes based on citation patterns or other factors, that we feel would be more accurate. Now I'll go out on a limb with an opinion that is counter to what I see in many papers. On the whole I think the ISI categories are pretty good. There are some categories that should be thrown out, some that should be split, some that could be merged, and some journals that are clearly mis-categorized, as mentioned in my recent Scientometrics [v64(3), p351-374] paper with Klavans and Borner. We can have lots of discussion about the details, but when you look at the whole thing, it's a good start. Perhaps I'm more comfortable than most with the ISI structure because I don't rely on it for evaluation. We use the emergent cluster structure from papers rather than any category or journal scheme. But that's another story. Can anyone recommend a better journal categorization scheme that is widely available? I'd love to hear of (and get a copy of) such a thing. Best regards, Kevin -----Original Message----- From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Loet Leydesdorff Sent: Saturday, March 04, 2006 6:15 AM To: SIGMETRICS at listserv.utk.edu Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Dear colleagues, The idea is interesting. However, there a few problems with this paper. First, the authors should not have used Pearson correlation coefficients to compare the rankings, but rank correlations (Spearman's rho or Kendall's tau). Second, it would have been interesting to have a rank correlation with "total cites" given recent discussions (Bensman). Third, the delineation of fields in terms of the ISI subject categories is very questionnable. However, the authors are very clear about their results: "We identified ... , but were unable to recognize a meaningful pattern in the results." (p. 9). I don't understand why one should then multiply the one measure with the other. What does multiplication to the error? Does one of you know a place where the ISI subject categories are justified? How are they produced? People seem to use them increasingly both in evaluation and research practices, but I have never been able to reproduce them using journal citation measures. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Eugene Garfield > Sent: Friday, March 03, 2006 6:37 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Further to yesterday's posting, "Prestige is factored into journal > ratings", here is another interesting and informative article > > FULL TEXT AVAILABLE AT : > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf > > email: {jbollen, marko, herbertv}@lanl.gov > > TITLE : Journal Status > > AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel > > SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Abstract > The status of an actor in a social context is commonly defined in > terms of two factors: the total number of endorsements the actor > receives from other actors and the prestige of the endorsing actors. > These two factors indicate the distinction between popularity and > expert appreciation of the actor, respectively. We refer to the former > as popularity and to the latter as prestige. These notions of > popularity and prestige also apply to the domain of scholarly > assessment. The ISI Impact Factor (ISI IF) is defined as the mean > number of citations a journal receives over a 2 year period. By merely > counting the amount of citations and disregarding the prestige of the > citing journals, the ISI IF is a metric of popularity, not of > prestige. We demonstrate how a weighted version of the popular > PageRank algorithm can be used to obtain a metric that reflects > prestige. We contrast the rankings of journals according to their ISI > IF and their weighted PageRank, and we provide an analysis that > reveals both significant overlaps and differences. > Furthermore, we introduce the Y-factor which is a simple combination > of both the ISI IF and the weighted PageRank, and find that the > resulting journal rankings correspond well to a general understanding > of journal status. > > > ______________________________________________ > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > FULL TEXT AVAILABLE AT : > http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR > http://guide.labanimal.com/news/2006/060213/full/439770a.html > > > Philip Ball : p.ball at nature.com > www.philipball.com > > Title: Prestige is factored into journal ratings > > Author(s): Ball P > > Source: NATURE 439 (7078): 770-771 FEB 16 2006 > > Document Type: News Item Language: English > Cited References: 0 Times Cited: 0 > > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 CRINAN ST, > LONDON > N1 9XW, ENGLAND > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA > > ISSN: 0028-0836 > From notsjb at LSU.EDU Mon Mar 6 12:41:27 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Mon, 6 Mar 2006 11:41:27 -0600 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: Loet, I see that you have once again taken my name in vain and again given me the opportunity to spout my ideas on SIGMETRICS. I must admit that I have not read the paper you discuss, because my doctor warned me against reading too many such papers, since I am fairly close to OD-ing on them. However, the conclusions you mention do seem a little peculiar. Due to detailed study of Gene Garfield's development and utilization of impact factor, I am coming to change my mind on this measure somewhat. It is for rather complicated reasons, which I shall try to explain below. In general I think that there is too much random error in citation data for the utilization of such precise techniques as correlation--Pearson, Spearman, whatever. Much results from exogenous citations due to an inability to define precise sets--a logical consequence of Bradford's Law of Scattering and Garfield's Law of Concentration. Impact factor suffers from a further source of error due to an inability to classify precisely sources into citable and non-citable--something which honest persons can disagree on. This inability severely affects the denominator of the impact factor equation. What is therefore needed is a technique that is crude and robust against such error. I have personally found it in the chi-square test of independence, which allows the conversion of citation measures into ordinal variables defined by broad categories. It also allows one to define the amount of error one is willing to accept, i.e., upper 10% vs. upper 25%. Use of this chi-square test may vindicate impact factor by demonstrating that it has the same strong relationship to expert ratings as do total citations. As a matter of fact, it may be a superior measure in that it will not only capture the importance of reseach journals but also of review journals. Close inspection of the top 10% of the journals recommended by the LSU chemistry faculty reveals it to be a balanced mix of research journals, review, journals, and the main teaching journal of chemistry. In other words, most facets of journal importance are captured by this measure, whereas total citations captures mainly research, and impact factor captures chiefly the review journals. However, broadening the categories may cause impact factor to capture both research and review though not the teaching facet. In any case I am going to test this in the revision of the JASIST paper I am now engaged in. Impact factor has the ability to do this for the very reasons Seglen denounces it. His main case against is based on the reasoning of the law of error and the role of the arithmetic mean in this law. This requires the normal distribution for the arithmetic mean to be an accurate estimate of central tendency. However, due to the highly skewed distributions with which we deal, the arithmetic mean is always way above the other estimates of central tendency such as the median or the geometric mean due to the high degree of variance caused the dominant observations. Seglen's reasoning collapses once one realizes that a journal's or scientist's importance is not measured by central tendency but by the variance caused by the few important articles published by the journal or scientist. Therefore, scientific importance is the result of variance and not central tendency. The arithmetic mean, which impact factor attempts to estimate, better captures the variance. To demonstrate, I have converted Garfield's constant for the year 1993 into binomial p and the Poisson lambda The way I did this is in the attached Excel file. You will see the binomial p is a lousy 0.0003, which converts into a Poisson lambda or Garfield's constant of 2.15 for the year. This is the probability or the rate articles were cited in 1993 on the assumption of probabilistic homogeneity. However, since there is probabilistic heterogeneity, most articles have to have a citation rate below Garfield's constant. True to form, of the 5000 journals covered that year, 4500 journals were below to Garfield's constant. 2.15 is an awful small range to squeeze 4500 journals into and expect meaningful quantitative distinctions. Utilization of a central tendency measure puts one right smack in the middle of that tight range. Small as this may be, the probabilities and lambda were actually much smaller, for Garfield's constant is based on the set of articles actually cited that year, i.e., it it truncated on the left and does not take into account the articles that could have been cited but were not. I do not have the technical or intellectual ability to estimate this zero class. I do know that Sir Maurice Kendall backed off from the problem when he confronted it in Bradford's Law, and who the hell am I compared to Maurice Kendall. I wish that somebody would write an article understandable to simpletons on how to make such estimates. From my perspective, this would be one of the most important articles ever written. Sorry for the tirade, but I thought I'd float a few trial balloons to be shot down. SB (See attached file: GarConst.xls) Loet Leydesdorff @LISTSERV.UTK.EDU> on 03/04/2006 07:14:57 AM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Dear colleagues, The idea is interesting. However, there a few problems with this paper. First, the authors should not have used Pearson correlation coefficients to compare the rankings, but rank correlations (Spearman's rho or Kendall's tau). Second, it would have been interesting to have a rank correlation with "total cites" given recent discussions (Bensman). Third, the delineation of fields in terms of the ISI subject categories is very questionnable. However, the authors are very clear about their results: "We identified ... , but were unable to recognize a meaningful pattern in the results." (p. 9). I don't understand why one should then multiply the one measure with the other. What does multiplication to the error? Does one of you know a place where the ISI subject categories are justified? How are they produced? People seem to use them increasingly both in evaluation and research practices, but I have never been able to reproduce them using journal citation measures. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Eugene Garfield > Sent: Friday, March 03, 2006 6:37 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and > Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Further to yesterday's posting, "Prestige is factored into > journal ratings", here is another interesting and informative article > > FULL TEXT AVAILABLE AT : > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf > > email: {jbollen, marko, herbertv}@lanl.gov > > TITLE : Journal Status > > AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel > > SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Abstract > The status of an actor in a social context is commonly > defined in terms of two factors: the total number of > endorsements the actor receives from other actors and the > prestige of the endorsing actors. These two factors indicate > the distinction between popularity and expert appreciation of > the actor, respectively. We refer to the former as popularity > and to the latter as prestige. These notions of popularity > and prestige also apply to the domain of scholarly > assessment. The ISI Impact Factor (ISI IF) is defined as the > mean number of citations a journal receives over a 2 year > period. By merely counting the amount of citations and > disregarding the prestige of the citing journals, the ISI IF > is a metric of popularity, not of prestige. We demonstrate > how a weighted version of the popular PageRank algorithm can > be used to obtain a metric that reflects prestige. We > contrast the rankings of journals according to their ISI IF > and their weighted PageRank, and we provide an analysis that > reveals both significant overlaps and differences. > Furthermore, we introduce the Y-factor which is a simple > combination of both the ISI IF and the weighted PageRank, and > find that the resulting journal rankings correspond well to a > general understanding of journal status. > > > ______________________________________________ > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > FULL TEXT AVAILABLE AT : > http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR > http://guide.labanimal.com/news/2006/060213/full/439770a.html > > > Philip Ball : p.ball at nature.com > www.philipball.com > > Title: Prestige is factored into journal ratings > > Author(s): Ball P > > Source: NATURE 439 (7078): 770-771 FEB 16 2006 > > Document Type: News Item Language: English > Cited References: 0 Times Cited: 0 > > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 > CRINAN ST, LONDON > N1 9XW, ENGLAND > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA > > ISSN: 0028-0836 > -------------- next part -------------- A non-text attachment was scrubbed... Name: GarConst.xls Type: application/msexcel Size: 17404 bytes Desc: not available URL: From loet at LEYDESDORFF.NET Mon Mar 6 14:53:48 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Mon, 6 Mar 2006 20:53:48 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: <46EAC19F3066C14BB20DF799A649C5F4020DB6B9@ES23SNLNT.srn.sandia.gov> Message-ID: Dear Kevin, The above figure is from my paper entitled Can Scientific Journals be Classified in terms of Aggregated Journal-Journal Citation Relations using the Journal Citation Reports? Journal of the American Society for Information Science and Technology (forthcoming). < pdf-version> . It made me less optimistic than you about the quality of the classification and I asked therefore one colleague at ISI how this classification was made, and he answered that he thought that there was no serious analytical basis for it. My question at this list was more to ask whether someone knew the basis of it than for being reassured. It seems very worrysome to me that this classification is increasingly used while it seems based on ....? I cannot find any confirmation. Your inference that the discussion about inclusion warrants the quality of the classification is also not warranted by research. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Boyack, Kevin W > Sent: Monday, March 06, 2006 4:42 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Hello everyone, > > Regarding ISI subject categories, I don't know how they are > picked, but you can read about journal inclusion at > http://scientific.thomson.com/free/essays/selectionofmaterial/ > journalsel > ection/. I would assume that if this much effort is going > into picking journals to index, there is at least some level > of thought going into subject categorization. After all, the > things ISI is looking at to decide on journal inclusion are > in many cases the same things you would look at for categorization. > > Now I won't claim that ISI's categorization is perfect, or > that it is updated as frequently as it might be, but I have > yet to see anyone put forth a better categorization scheme > that is widely available. I'm sure there are many of us that > either have, or could easily, make up our own schemes based > on citation patterns or other factors, that we feel would be > more accurate. > > Now I'll go out on a limb with an opinion that is counter to > what I see in many papers. On the whole I think the ISI > categories are pretty good. > There are some categories that should be thrown out, some > that should be split, some that could be merged, and some > journals that are clearly mis-categorized, as mentioned in my > recent Scientometrics [v64(3), p351-374] paper with Klavans > and Borner. We can have lots of discussion about the details, > but when you look at the whole thing, it's a good start. > > Perhaps I'm more comfortable than most with the ISI structure > because I don't rely on it for evaluation. We use the > emergent cluster structure from papers rather than any > category or journal scheme. But that's another story. > > Can anyone recommend a better journal categorization scheme > that is widely available? I'd love to hear of (and get a copy > of) such a thing. > > Best regards, > Kevin > > > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Loet Leydesdorff > Sent: Saturday, March 04, 2006 6:15 AM > To: SIGMETRICS at listserv.utk.edu > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Dear colleagues, > > The idea is interesting. However, there a few problems with > this paper. > First, the authors should not have used Pearson correlation > coefficients to compare the rankings, but rank correlations > (Spearman's rho or Kendall's tau). Second, it would have been > interesting to have a rank correlation with "total cites" > given recent discussions (Bensman). > Third, the delineation of fields in terms of the ISI subject > categories is very questionnable. > > However, the authors are very clear about their results: "We > identified ... > , but were unable to recognize a meaningful pattern in the > results." (p. > 9). > I don't understand why one should then multiply the one > measure with the other. What does multiplication to the error? > > Does one of you know a place where the ISI subject categories > are justified? > How are they produced? People seem to use them increasingly > both in evaluation and research practices, but I have never > been able to reproduce them using journal citation measures. > > With best wishes, > > > Loet > > > ________________________________ > Loet Leydesdorff > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal 48, 1012 CX Amsterdam. > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; http://www.leydesdorff.net/ > > > > > -----Original Message----- > > From: ASIS&T Special Interest Group on Metrics > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Eugene Garfield > > Sent: Friday, March 03, 2006 6:37 PM > > To: SIGMETRICS at LISTSERV.UTK.EDU > > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > Further to yesterday's posting, "Prestige is factored into journal > > ratings", here is another interesting and informative article > > > > FULL TEXT AVAILABLE AT : > > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf > > > > email: {jbollen, marko, herbertv}@lanl.gov > > > > TITLE : Journal Status > > > > AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert > Van de Sompel > > > > SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > Abstract > > The status of an actor in a social context is commonly defined in > > terms of two factors: the total number of endorsements the actor > > receives from other actors and the prestige of the endorsing actors. > > These two factors indicate the distinction between popularity and > > expert appreciation of the actor, respectively. We refer to > the former > > > as popularity and to the latter as prestige. These notions of > > popularity and prestige also apply to the domain of scholarly > > assessment. The ISI Impact Factor (ISI IF) is defined as the mean > > number of citations a journal receives over a 2 year > period. By merely > > > counting the amount of citations and disregarding the > prestige of the > > citing journals, the ISI IF is a metric of popularity, not of > > prestige. We demonstrate how a weighted version of the popular > > PageRank algorithm can be used to obtain a metric that reflects > > prestige. We contrast the rankings of journals according to > their ISI > > IF and their weighted PageRank, and we provide an analysis that > > reveals both significant overlaps and differences. > > Furthermore, we introduce the Y-factor which is a simple > combination > > of both the ISI IF and the weighted PageRank, and find that the > > resulting journal rankings correspond well to a general > understanding > > of journal status. > > > > > > ______________________________________________ > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > FULL TEXT AVAILABLE AT : > > > http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR > > http://guide.labanimal.com/news/2006/060213/full/439770a.html > > > > > > Philip Ball : p.ball at nature.com > > www.philipball.com > > > > Title: Prestige is factored into journal ratings > > > > Author(s): Ball P > > > > Source: NATURE 439 (7078): 770-771 FEB 16 2006 > > > > Document Type: News Item Language: English > > Cited References: 0 Times Cited: 0 > > > > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 > CRINAN ST, > > LONDON > > N1 9XW, ENGLAND > > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA > > > > ISSN: 0028-0836 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fig12.jpg Type: image/jpeg Size: 36597 bytes Desc: not available URL: From loet at LEYDESDORFF.NET Mon Mar 6 16:12:54 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Mon, 6 Mar 2006 22:12:54 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: Dear Stephen, I apologize if I dragged you into the discussion with improper argument, but I did not want to mention your idea of using total cites as an indicator without providing a proper reference. The reasoning in your posting is difficult for me to follow, but I look forward to reading the full paper. My experience is that reading the full paper, one begins to understand. I found your previous argument about using total cites very convincing because of its high correlation with faculty ratings and its orthogonality to the impact factor. It seemed to me that the impact factor measures something very different from the prestige of a journal. Figure 1: Component plot in rotated space (sources: JCR, 1993; Bensman, 2001; forthcoming; Bensman & Wilder, 1998). From: Visualization of the Citation Impact Environments of Scientific Journals: An online mapping exercise, Journal of the American Society for Information Science and Technology (forthcoming). . > With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman > Sent: Monday, March 06, 2006 6:41 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > Loet, > I see that you have once again taken my name in vain and > again given me the opportunity to spout my ideas on > SIGMETRICS. I must admit that I have not read the paper you > discuss, because my doctor warned me against reading too many > such papers, since I am fairly close to OD-ing on them. > However, the conclusions you mention do seem a little peculiar. > > Due to detailed study of Gene Garfield's development and > utilization of impact factor, I am coming to change my mind > on this measure somewhat. It is for rather complicated > reasons, which I shall try to explain below. > > In general I think that there is too much random error in > citation data for the utilization of such precise techniques > as correlation--Pearson, Spearman, whatever. Much results > from exogenous citations due to an inability to define > precise sets--a logical consequence of Bradford's Law of > Scattering and Garfield's Law of Concentration. Impact > factor suffers from a further source of error due to an > inability to classify precisely sources into citable and > non-citable--something which honest persons can disagree on. > This inability severely affects the denominator of the impact > factor equation. What is therefore needed is a technique > that is crude and robust against such error. I have > personally found it in the chi-square test of independence, > which allows the conversion of citation measures into ordinal > variables defined by broad categories. It also allows one to > define the amount of error one is willing to accept, i.e., > upper 10% vs. > upper 25%. > > Use of this chi-square test may vindicate impact factor by > demonstrating that it has the same strong relationship to > expert ratings as do total citations. As a matter of fact, > it may be a superior measure in that it will not only capture > the importance of reseach journals but also of review > journals. Close inspection of the top 10% of the journals > recommended by the LSU chemistry faculty reveals it to be a > balanced mix of research journals, review, journals, and the > main teaching journal of chemistry. > In other words, most facets of journal importance are > captured by this measure, whereas total citations captures > mainly research, and impact factor captures chiefly the > review journals. However, broadening the categories may > cause impact factor to capture both research and review > though not the teaching facet. In any case I am going to > test this in the revision of the JASIST paper I am now engaged in. > > Impact factor has the ability to do this for the very reasons > Seglen denounces it. His main case against is based on the > reasoning of the law of error and the role of the arithmetic > mean in this law. This requires the normal distribution for > the arithmetic mean to be an accurate estimate of central > tendency. However, due to the highly skewed distributions > with which we deal, the arithmetic mean is always way above > the other estimates of central tendency such as the median or > the geometric mean due to the high degree of variance caused > the dominant observations. Seglen's reasoning collapses once > one realizes that a journal's or scientist's importance is > not measured by central tendency but by the variance caused > by the few important articles published by the journal or scientist. > Therefore, scientific importance is the result of variance > and not central tendency. The arithmetic mean, which impact > factor attempts to estimate, better captures the variance. > > To demonstrate, I have converted Garfield's constant for the > year 1993 into binomial p and the Poisson lambda The way I > did this is in the attached Excel file. You will see the > binomial p is a lousy 0.0003, which converts into a Poisson > lambda or Garfield's constant of 2.15 for the year. This is > the probability or the rate articles were cited in 1993 on > the assumption of probabilistic homogeneity. However, since > there is probabilistic heterogeneity, most articles have to > have a citation rate below Garfield's constant. True to > form, of the 5000 journals covered that year, 4500 journals > were below to Garfield's constant. 2.15 is an awful small > range to squeeze 4500 journals into and expect meaningful > quantitative distinctions. Utilization of a central tendency > measure puts one right smack in the middle of that tight > range. Small as this may be, the probabilities and lambda > were actually much smaller, for Garfield's constant is based > on the set of articles actually cited that year, i.e., it it > truncated on the left and does not take into account the > articles that could have been cited but were not. I do not > have the technical or intellectual ability to estimate this > zero class. I do know that Sir Maurice Kendall backed off > from the problem when he confronted it in Bradford's Law, and > who the hell am I compared to Maurice Kendall. I wish that > somebody would write an article understandable to simpletons > on how to make such estimates. From my perspective, this > would be one of the most important articles ever written. > > Sorry for the tirade, but I thought I'd float a few trial > balloons to be shot down. > > SB > > (See attached file: GarConst.xls) > > > > > > > > > Loet Leydesdorff @LISTSERV.UTK.EDU> on > 03/04/2006 > 07:14:57 AM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > Rodriguez, and Herbert > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Dear colleagues, > > The idea is interesting. However, there a few problems with > this paper. > First, the authors should not have used Pearson correlation > coefficients to compare the rankings, but rank correlations > (Spearman's rho or Kendall's tau). Second, it would have been > interesting to have a rank correlation with "total cites" > given recent discussions (Bensman). Third, the delineation of > fields in terms of the ISI subject categories is very questionnable. > > However, the authors are very clear about their results: "We > identified ... > , but were unable to recognize a meaningful pattern in the > results." (p. > 9). > I don't understand why one should then multiply the one > measure with the other. What does multiplication to the error? > > Does one of you know a place where the ISI subject categories > are justified? > How are they produced? People seem to use them increasingly > both in evaluation and research practices, but I have never > been able to reproduce them using journal citation measures. > > With best wishes, > > > Loet > > > ________________________________ > Loet Leydesdorff > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal 48, 1012 CX Amsterdam. > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; http://www.leydesdorff.net/ > > > > > -----Original Message----- > > From: ASIS&T Special Interest Group on Metrics > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Eugene Garfield > > Sent: Friday, March 03, 2006 6:37 PM > > To: SIGMETRICS at LISTSERV.UTK.EDU > > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > Further to yesterday's posting, "Prestige is factored into journal > > ratings", here is another interesting and informative article > > > > FULL TEXT AVAILABLE AT : > > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf > > > > email: {jbollen, marko, herbertv}@lanl.gov > > > > TITLE : Journal Status > > > > AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert > Van de Sompel > > > > SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > Abstract > > The status of an actor in a social context is commonly defined in > > terms of two factors: the total number of endorsements the actor > > receives from other actors and the prestige of the > endorsing actors. > > These two factors indicate the distinction between popularity and > > expert appreciation of the actor, respectively. We refer to > the former > > as popularity and to the latter as prestige. These notions of > > popularity and prestige also apply to the domain of scholarly > > assessment. The ISI Impact Factor (ISI IF) is defined as the mean > > number of citations a journal receives over a 2 year > period. By merely > > counting the amount of citations and disregarding the > prestige of the > > citing journals, the ISI IF is a metric of popularity, not of > > prestige. We demonstrate how a weighted version of the popular > > PageRank algorithm can be used to obtain a metric that reflects > > prestige. We contrast the rankings of journals according to > their ISI > > IF and their weighted PageRank, and we provide an analysis that > > reveals both significant overlaps and differences. > > Furthermore, we introduce the Y-factor which is a simple > combination > > of both the ISI IF and the weighted PageRank, and find that the > > resulting journal rankings correspond well to a general > understanding > > of journal status. > > > > > > ______________________________________________ > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > FULL TEXT AVAILABLE AT : > > > http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR > > http://guide.labanimal.com/news/2006/060213/full/439770a.html > > > > > > Philip Ball : p.ball at nature.com > > www.philipball.com > > > > Title: Prestige is factored into journal ratings > > > > Author(s): Ball P > > > > Source: NATURE 439 (7078): 770-771 FEB 16 2006 > > > > Document Type: News Item Language: English > > Cited References: 0 Times Cited: 0 > > > > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 > CRINAN ST, > > LONDON > > N1 9XW, ENGLAND > > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA > > > > ISSN: 0028-0836 > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: clip_image002.gif Type: image/gif Size: 3683 bytes Desc: not available URL: From kboyack at SANDIA.GOV Mon Mar 6 17:01:23 2006 From: kboyack at SANDIA.GOV (Boyack, Kevin W) Date: Mon, 6 Mar 2006 15:01:23 -0700 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: Dear Loet, Your choice of category to highlight provides for some very interesting points. "Biochemistry and Molecular Biology" is a very large and diverse grouping in the ISI data. Journals are jointly categorized in this category with some 59 other categories, which fact by itself says that this category is too broad - thus supporting your argument. The largest multiple categorizations are (2002 data) with: CELL BIOLOGY 71 BIOPHYSICS 45 BIOTECH & APPLIED MICRO 24 GENETICS & HEREDITY 23 PHARMACOLOGY & PHARMACY 23 BIOCHEM RESEARCH METHODS 16 ENDOCRINOLOGY & METABOLISM 15 PLANT SCIENCES 13 IMMUNOLOGY 11 and so on. But this also says to me that your comparison below is showing a bit of "apples and oranges". For instance, one would not expect journals that are jointly categorized BIOCHEM / BIOPHYSICS or PHARMA or PLANT SCIENCES ... to appear in the overlap set of your venn diagrams. Your factor analysis categorizations would have put these journals in factors different than those you highlight below (JACS-related and JBC-related), and it is not entirely fair to include them. If you were to limit your ISI comparison set to journals either singly categorized as BIOCHEM or to those plus journals doubly categorized to BIOCHEM + CELL BIOLOGY, your pictures would look much different. I would love to see that analysis. When I look at the distribution of the ISI categories in my cluster maps, and see that, for the most part, 80% of the members of each cluster belong to a dominant ISI category, I continue to retain my optimism. Regarding how categories are assigned - I am perhaps too much of an optimist, but I have the hope that whoever is doing it is smart, and is relying on some data to make judgments. You are right about the pitfalls of using this categorization blindly - I share your concerns. But people will increasingly use it until something is shown to be better, and becomes readily available and publicly accepted. Your local citation environments are a great step forward. I'd love to see them in more common use. Best regards, Kevin ________________________________ From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Loet Leydesdorff Sent: Monday, March 06, 2006 12:54 PM To: SIGMETRICS at LISTSERV.UTK.EDU Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Dear Kevin, The above figure is from my paper entitled Can Scientific Journals be Classified in terms of Aggregated Journal-Journal Citation Relations using the Journal Citation Reports? Journal of the American Society for Information Science and Technology (forthcoming). < pdf-version > . It made me less optimistic than you about the quality of the classification and I asked therefore one colleague at ISI how this classification was made, and he answered that he thought that there was no serious analytical basis for it. My question at this list was more to ask whether someone knew the basis of it than for being reassured. It seems very worrysome to me that this classification is increasingly used while it seems based on ....? I cannot find any confirmation. Your inference that the discussion about inclusion warrants the quality of the classification is also not warranted by research. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Boyack, Kevin W > Sent: Monday, March 06, 2006 4:42 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Hello everyone, > > Regarding ISI subject categories, I don't know how they are > picked, but you can read about journal inclusion at > http://scientific.thomson.com/free/essays/selectionofmaterial/ > journalsel > ection/. I would assume that if this much effort is going > into picking journals to index, there is at least some level > of thought going into subject categorization. After all, the > things ISI is looking at to decide on journal inclusion are > in many cases the same things you would look at for categorization. > > Now I won't claim that ISI's categorization is perfect, or > that it is updated as frequently as it might be, but I have > yet to see anyone put forth a better categorization scheme > that is widely available. I'm sure there are many of us that > either have, or could easily, make up our own schemes based > on citation patterns or other factors, that we feel would be > more accurate. > > Now I'll go out on a limb with an opinion that is counter to > what I see in many papers. On the whole I think the ISI > categories are pretty good. > There are some categories that should be thrown out, some > that should be split, some that could be merged, and some > journals that are clearly mis-categorized, as mentioned in my > recent Scientometrics [v64(3), p351-374] paper with Klavans > and Borner. We can have lots of discussion about the details, > but when you look at the whole thing, it's a good start. > > Perhaps I'm more comfortable than most with the ISI structure > because I don't rely on it for evaluation. We use the > emergent cluster structure from papers rather than any > category or journal scheme. But that's another story. > > Can anyone recommend a better journal categorization scheme > that is widely available? I'd love to hear of (and get a copy > of) such a thing. > > Best regards, > Kevin > > > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Loet Leydesdorff > Sent: Saturday, March 04, 2006 6:15 AM > To: SIGMETRICS at listserv.utk.edu > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Dear colleagues, > > The idea is interesting. However, there a few problems with > this paper. > First, the authors should not have used Pearson correlation > coefficients to compare the rankings, but rank correlations > (Spearman's rho or Kendall's tau). Second, it would have been > interesting to have a rank correlation with "total cites" > given recent discussions (Bensman). > Third, the delineation of fields in terms of the ISI subject > categories is very questionnable. > > However, the authors are very clear about their results: "We > identified ... > , but were unable to recognize a meaningful pattern in the > results." (p. > 9). > I don't understand why one should then multiply the one > measure with the other. What does multiplication to the error? > > Does one of you know a place where the ISI subject categories > are justified? > How are they produced? People seem to use them increasingly > both in evaluation and research practices, but I have never > been able to reproduce them using journal citation measures. > > With best wishes, > > > Loet > > > ________________________________ > Loet Leydesdorff > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal 48, 1012 CX Amsterdam. > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; http://www.leydesdorff.net/ > > > > > -----Original Message----- > > From: ASIS&T Special Interest Group on Metrics > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Eugene Garfield > > Sent: Friday, March 03, 2006 6:37 PM > > To: SIGMETRICS at LISTSERV.UTK.EDU > > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > Further to yesterday's posting, "Prestige is factored into journal > > ratings", here is another interesting and informative article > > > > FULL TEXT AVAILABLE AT : > > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf > > > > email: {jbollen, marko, herbertv}@lanl.gov > > > > TITLE : Journal Status > > > > AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert > Van de Sompel > > > > SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > Abstract > > The status of an actor in a social context is commonly defined in > > terms of two factors: the total number of endorsements the actor > > receives from other actors and the prestige of the endorsing actors. > > These two factors indicate the distinction between popularity and > > expert appreciation of the actor, respectively. We refer to > the former > > > as popularity and to the latter as prestige. These notions of > > popularity and prestige also apply to the domain of scholarly > > assessment. The ISI Impact Factor (ISI IF) is defined as the mean > > number of citations a journal receives over a 2 year > period. By merely > > > counting the amount of citations and disregarding the > prestige of the > > citing journals, the ISI IF is a metric of popularity, not of > > prestige. We demonstrate how a weighted version of the popular > > PageRank algorithm can be used to obtain a metric that reflects > > prestige. We contrast the rankings of journals according to > their ISI > > IF and their weighted PageRank, and we provide an analysis that > > reveals both significant overlaps and differences. > > Furthermore, we introduce the Y-factor which is a simple > combination > > of both the ISI IF and the weighted PageRank, and find that the > > resulting journal rankings correspond well to a general > understanding > > of journal status. > > > > > > ______________________________________________ > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > FULL TEXT AVAILABLE AT : > > > http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR > > http://guide.labanimal.com/news/2006/060213/full/439770a.html > > > > > > Philip Ball : p.ball at nature.com > > www.philipball.com > > > > Title: Prestige is factored into journal ratings > > > > Author(s): Ball P > > > > Source: NATURE 439 (7078): 770-771 FEB 16 2006 > > > > Document Type: News Item Language: English > > Cited References: 0 Times Cited: 0 > > > > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 > CRINAN ST, > > LONDON > > N1 9XW, ENGLAND > > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA > > > > ISSN: 0028-0836 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fig12.jpg Type: image/jpeg Size: 36597 bytes Desc: fig12.jpg URL: From notsjb at LSU.EDU Mon Mar 6 17:35:10 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Mon, 6 Mar 2006 16:35:10 -0600 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: Loet, You and I are perhaps looking at two different aspects of the problem. I am looking at it from the perspective of a librarian trying to decide which journals should be provided with permanent access on a subscription basis and which should be accessed through some form of intermittent document delivery. Therefore, I am interested not only in prestige but also in functionality, i.e., what function does the journal serve--reporting of research, reviewing of literature, assistance in teaching, provision of current news, etc. Citation measures either capture one facet of functionality--total citations seem to capture reporting of research, impact factor, review literature--or fail to capture the functionality at all, i.e., teaching or reporting of current news. Total citations cannot capture the review literature, because review journals are usually very small even though highly rated by scientists, but since impact factor captures both review literature and current research significance--which is usually the same as historical historical research significance due the stability of patterns--and the correlation of total citations with impact factor is high enough so that journals high on both can be captured in a broad category robust against random error, it seems to me that impact factor can capture two facets of functionality unlike total citations, which can only capture one. The hypothesis remains to be tested. Prestige appears to operate separately from functionality. The greatest cause of variance in all four measures is their belonging to the category of US association journals. The journals of the American Chemical Society are dominant on all four measures. Through various evaluations of US research-doctorate programs by peer ratings and citations, I can trace this dominance to scientists employed by the traditionally elite US research insitutions. Thus, variance and prestige in all four measures is a function of the social stratification system of US scientific institutions. There remains the question of how do foreign scientists relate to the US social stratification system. If they form a part of it, the foreigners can use ISI citations for evaluation and other puposes. If they do not form part of it, then foreigners using ISI citations may only be rating themselves by how much their work is being accepted by the scientists within this system. I have no answer to this question. I suppose that now you are more confused than ever. I did my best, but it is complicated as all hell. SB Loet Leydesdorff @listserv.utk.edu> on 03/06/2006 03:12:54 PM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at listserv.utk.edu cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Dear Stephen, I apologize if I dragged you into the discussion with improper argument, but I did not want to mention your idea of using total cites as an indicator without providing a proper reference. The reasoning in your posting is difficult for me to follow, but I look forward to reading the full paper. My experience is that reading the full paper, one begins to understand. I found your previous argument about using total cites very convincing because of its high correlation with faculty ratings and its orthogonality to the impact factor. It seemed to me that the impact factor measures something very different from the prestige of a journal. (Embedded image moved to file: pic17086.gif) Figure 1: Component plot in rotated space (sources: JCR, 1993; Bensman, 2001; forthcoming; Bensman & Wilder, 1998). From: Visualization of the Citation Impact Environments of Scientific Journals: An online mapping exercise, Journal of the American Society for Information Science and Technology (forthcoming). . With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman > Sent: Monday, March 06, 2006 6:41 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > Loet, > I see that you have once again taken my name in vain and > again given me the opportunity to spout my ideas on > SIGMETRICS. I must admit that I have not read the paper you > discuss, because my doctor warned me against reading too many > such papers, since I am fairly close to OD-ing on them. > However, the conclusions you mention do seem a little peculiar. > > Due to detailed study of Gene Garfield's development and > utilization of impact factor, I am coming to change my mind > on this measure somewhat. It is for rather complicated > reasons, which I shall try to explain below. > > In general I think that there is too much random error in > citation data for the utilization of such precise techniques > as correlation--Pearson, Spearman, whatever. Much results > from exogenous citations due to an inability to define > precise sets--a logical consequence of Bradford's Law of > Scattering and Garfield's Law of Concentration. Impact > factor suffers from a further source of error due to an > inability to classify precisely sources into citable and > non-citable--something which honest persons can disagree on. > This inability severely affects the denominator of the impact > factor equation. What is therefore needed is a technique > that is crude and robust against such error. I have > personally found it in the chi-square test of independence, > which allows the conversion of citation measures into ordinal > variables defined by broad categories. It also allows one to > define the amount of error one is willing to accept, i.e., > upper 10% vs. > upper 25%. > > Use of this chi-square test may vindicate impact factor by > demonstrating that it has the same strong relationship to > expert ratings as do total citations. As a matter of fact, > it may be a superior measure in that it will not only capture > the importance of reseach journals but also of review > journals. Close inspection of the top 10% of the journals > recommended by the LSU chemistry faculty reveals it to be a > balanced mix of research journals, review, journals, and the > main teaching journal of chemistry. > In other words, most facets of journal importance are > captured by this measure, whereas total citations captures > mainly research, and impact factor captures chiefly the > review journals. However, broadening the categories may > cause impact factor to capture both research and review > though not the teaching facet. In any case I am going to > test this in the revision of the JASIST paper I am now engaged in. > > Impact factor has the ability to do this for the very reasons > Seglen denounces it. His main case against is based on the > reasoning of the law of error and the role of the arithmetic > mean in this law. This requires the normal distribution for > the arithmetic mean to be an accurate estimate of central > tendency. However, due to the highly skewed distributions > with which we deal, the arithmetic mean is always way above > the other estimates of central tendency such as the median or > the geometric mean due to the high degree of variance caused > the dominant observations. Seglen's reasoning collapses once > one realizes that a journal's or scientist's importance is > not measured by central tendency but by the variance caused > by the few important articles published by the journal or scientist. > Therefore, scientific importance is the result of variance > and not central tendency. The arithmetic mean, which impact > factor attempts to estimate, better captures the variance. > > To demonstrate, I have converted Garfield's constant for the > year 1993 into binomial p and the Poisson lambda The way I > did this is in the attached Excel file. You will see the > binomial p is a lousy 0.0003, which converts into a Poisson > lambda or Garfield's constant of 2.15 for the year. This is > the probability or the rate articles were cited in 1993 on > the assumption of probabilistic homogeneity. However, since > there is probabilistic heterogeneity, most articles have to > have a citation rate below Garfield's constant. True to > form, of the 5000 journals covered that year, 4500 journals > were below to Garfield's constant. 2.15 is an awful small > range to squeeze 4500 journals into and expect meaningful > quantitative distinctions. Utilization of a central tendency > measure puts one right smack in the middle of that tight > range. Small as this may be, the probabilities and lambda > were actually much smaller, for Garfield's constant is based > on the set of articles actually cited that year, i.e., it it > truncated on the left and does not take into account the > articles that could have been cited but were not. I do not > have the technical or intellectual ability to estimate this > zero class. I do know that Sir Maurice Kendall backed off > from the problem when he confronted it in Bradford's Law, and > who the hell am I compared to Maurice Kendall. I wish that > somebody would write an article understandable to simpletons > on how to make such estimates. From my perspective, this > would be one of the most important articles ever written. > > Sorry for the tirade, but I thought I'd float a few trial > balloons to be shot down. > > SB > > (See attached file: GarConst.xls) > > > > > > > > > Loet Leydesdorff @LISTSERV.UTK.EDU> on > 03/04/2006 > 07:14:57 AM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > Rodriguez, and Herbert > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Dear colleagues, > > The idea is interesting. However, there a few problems with > this paper. > First, the authors should not have used Pearson correlation > coefficients to compare the rankings, but rank correlations > (Spearman's rho or Kendall's tau). Second, it would have been > interesting to have a rank correlation with "total cites" > given recent discussions (Bensman). Third, the delineation of > fields in terms of the ISI subject categories is very questionnable. > > However, the authors are very clear about their results: "We > identified ... > , but were unable to recognize a meaningful pattern in the > results." (p. > 9). > I don't understand why one should then multiply the one > measure with the other. What does multiplication to the error? > > Does one of you know a place where the ISI subject categories > are justified? > How are they produced? People seem to use them increasingly > both in evaluation and research practices, but I have never > been able to reproduce them using journal citation measures. > > With best wishes, > > > Loet > > > ________________________________ > Loet Leydesdorff > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal 48, 1012 CX Amsterdam. > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; http://www.leydesdorff.net/ > > > > > -----Original Message----- > > From: ASIS&T Special Interest Group on Metrics > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Eugene Garfield > > Sent: Friday, March 03, 2006 6:37 PM > > To: SIGMETRICS at LISTSERV.UTK.EDU > > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > Further to yesterday's posting, "Prestige is factored into journal > > ratings", here is another interesting and informative article > > > > FULL TEXT AVAILABLE AT : > > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf > > > > email: {jbollen, marko, herbertv}@lanl.gov > > > > TITLE : Journal Status > > > > AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert > Van de Sompel > > > > SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > Abstract > > The status of an actor in a social context is commonly defined in > > terms of two factors: the total number of endorsements the actor > > receives from other actors and the prestige of the > endorsing actors. > > These two factors indicate the distinction between popularity and > > expert appreciation of the actor, respectively. We refer to > the former > > as popularity and to the latter as prestige. These notions of > > popularity and prestige also apply to the domain of scholarly > > assessment. The ISI Impact Factor (ISI IF) is defined as the mean > > number of citations a journal receives over a 2 year > period. By merely > > counting the amount of citations and disregarding the > prestige of the > > citing journals, the ISI IF is a metric of popularity, not of > > prestige. We demonstrate how a weighted version of the popular > > PageRank algorithm can be used to obtain a metric that reflects > > prestige. We contrast the rankings of journals according to > their ISI > > IF and their weighted PageRank, and we provide an analysis that > > reveals both significant overlaps and differences. > > Furthermore, we introduce the Y-factor which is a simple > combination > > of both the ISI IF and the weighted PageRank, and find that the > > resulting journal rankings correspond well to a general > understanding > > of journal status. > > > > > > ______________________________________________ > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > FULL TEXT AVAILABLE AT : > > > http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR > > http://guide.labanimal.com/news/2006/060213/full/439770a.html > > > > > > Philip Ball : p.ball at nature.com > > www.philipball.com > > > > Title: Prestige is factored into journal ratings > > > > Author(s): Ball P > > > > Source: NATURE 439 (7078): 770-771 FEB 16 2006 > > > > Document Type: News Item Language: English > > Cited References: 0 Times Cited: 0 > > > > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 > CRINAN ST, > > LONDON > > N1 9XW, ENGLAND > > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA > > > > ISSN: 0028-0836 > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: pic17086.gif Type: image/gif Size: 3683 bytes Desc: not available URL: From dgoodman at PRINCETON.EDU Tue Mar 7 01:09:09 2006 From: dgoodman at PRINCETON.EDU (David Goodman) Date: Tue, 7 Mar 2006 01:09:09 -0500 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: Dear Stephen, Surely there are two different purposes intended: Local citations (and local generation's) is what a librarian must use in evaluating journals for his local library (as well as other things, of course, like value for money.) They are also what one needs in doing "micro-bibliometrics" of patterns within a university or similar setting. But in evaluating journals for "quality" or intrinsic worth, or even value for money in a global sense, the conventional IFs would seem more effective. There might be some atypical situations where the "high ranking" authors used predominantly one set of journals, and the "lower ranking" authors used a different set, but I cannot recall a published example. But I do not agree that total citations is generally useful. Total citations A large journal with a low IF for its field and hight total citations is a repository for second rate articles, A small journal with the same relatively low impact factor might be a smaller repository for second-rate articles, or might be a very specialized good journal. Looking at which journals cite them will distinguish. There are however other good reasons to look at size such as examining the relationship between price and cost, as Bergstrom et al have done. A better global factor will be total number of generation's, when such data becomes available. From my experience in COUNTER, this will not be for quite a while. (I say "my experience" ; I do not speak for Counter.) There is a major technical problem: The numbers are not comparable between publishers, only between the journals of a single publisher--this has been know for about a year now, and the cause remains undetermined. See (Note that this also affects the usefulness of comparing generation's for local use; there is an option of examining logs, but that only measures the portion that goes through the portal; I do not know if anyone has calibrated this ratio locally--it will of course be vary by journal and by different universities. Once this is figured out, we will need to get the publishers to release global use figures. (Or we will need to get all universities willing to provide their calibrated portal count, less accurate though that may be.) Neither will be easy, and anyone interested in strategy can write to me off-list. What we will soon have, is the ability to provide global counts of all uses in IRs and possibly some other repositories, aggregated by journal. If OA comes soon enough, we won't need anything else. It's possible that we might get 100% OA before we have 100% publishers' statistics. Prestige is easy conceptually, though hard to measure. Imho, Prestige is the previous generation's use and IF. JCR has been around long enough for us to start doing longitudinal studies of the necessary time span. Dr. David Goodman Associate Professor Palmer School of Library and Information Science Long Island University and formerly Princeton University Library dgoodman at liu.edu dgoodman at princeton.edu From loet at LEYDESDORFF.NET Tue Mar 7 02:20:29 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Tue, 7 Mar 2006 08:20:29 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: <46EAC19F3066C14BB20DF799A649C5F4020DB6C2@ES23SNLNT.srn.sandia.gov> Message-ID: Dear Kevin, My only point was that I would like to know how these categories are assigned. I agree that some of these assigments are better than others and that it is a difficult job, but we cannot control anything until we know the how. Is there an algorithm, specific expertise or is it just done by one or more indexers? If you look for a better classification, one could use the one of the Library of Congress. That is done very professionally. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ _____ From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Boyack, Kevin W Sent: Monday, March 06, 2006 11:01 PM To: SIGMETRICS at LISTSERV.UTK.EDU Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Dear Loet, Your choice of category to highlight provides for some very interesting points. "Biochemistry and Molecular Biology" is a very large and diverse grouping in the ISI data. Journals are jointly categorized in this category with some 59 other categories, which fact by itself says that this category is too broad - thus supporting your argument. The largest multiple categorizations are (2002 data) with: CELL BIOLOGY 71 BIOPHYSICS 45 BIOTECH & APPLIED MICRO 24 GENETICS & HEREDITY 23 PHARMACOLOGY & PHARMACY 23 BIOCHEM RESEARCH METHODS 16 ENDOCRINOLOGY & METABOLISM 15 PLANT SCIENCES 13 IMMUNOLOGY 11 and so on. But this also says to me that your comparison below is showing a bit of "apples and oranges". For instance, one would not expect journals that are jointly categorized BIOCHEM / BIOPHYSICS or PHARMA or PLANT SCIENCES ... to appear in the overlap set of your venn diagrams. Your factor analysis categorizations would have put these journals in factors different than those you highlight below (JACS-related and JBC-related), and it is not entirely fair to include them. If you were to limit your ISI comparison set to journals either singly categorized as BIOCHEM or to those plus journals doubly categorized to BIOCHEM + CELL BIOLOGY, your pictures would look much different. I would love to see that analysis. When I look at the distribution of the ISI categories in my cluster maps, and see that, for the most part, 80% of the members of each cluster belong to a dominant ISI category, I continue to retain my optimism. Regarding how categories are assigned - I am perhaps too much of an optimist, but I have the hope that whoever is doing it is smart, and is relying on some data to make judgments. You are right about the pitfalls of using this categorization blindly - I share your concerns. But people will increasingly use it until something is shown to be better, and becomes readily available and publicly accepted. Your local citation environments are a great step forward. I'd love to see them in more common use. Best regards, Kevin _____ From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Loet Leydesdorff Sent: Monday, March 06, 2006 12:54 PM To: SIGMETRICS at LISTSERV.UTK.EDU Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Dear Kevin, The above figure is from my paper entitled Can Scientific Journals be Classified in terms of Aggregated Journal-Journal Citation Relations using the Journal Citation Reports? Journal of the American Society for Information Science and Technology (forthcoming). < pdf-version> . It made me less optimistic than you about the quality of the classification and I asked therefore one colleague at ISI how this classification was made, and he answered that he thought that there was no serious analytical basis for it. My question at this list was more to ask whether someone knew the basis of it than for being reassured. It seems very worrysome to me that this classification is increasingly used while it seems based on ....? I cannot find any confirmation. Your inference that the discussion about inclusion warrants the quality of the classification is also not warranted by research. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Boyack, Kevin W > Sent: Monday, March 06, 2006 4:42 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Hello everyone, > > Regarding ISI subject categories, I don't know how they are > picked, but you can read about journal inclusion at > http://scientific.thomson.com/free/essays/selectionofmaterial/ > journalsel > ection/. I would assume that if this much effort is going > into picking journals to index, there is at least some level > of thought going into subject categorization. After all, the > things ISI is looking at to decide on journal inclusion are > in many cases the same things you would look at for categorization. > > Now I won't claim that ISI's categorization is perfect, or > that it is updated as frequently as it might be, but I have > yet to see anyone put forth a better categorization scheme > that is widely available. I'm sure there are many of us that > either have, or could easily, make up our own schemes based > on citation patterns or other factors, that we feel would be > more accurate. > > Now I'll go out on a limb with an opinion that is counter to > what I see in many papers. On the whole I think the ISI > categories are pretty good. > There are some categories that should be thrown out, some > that should be split, some that could be merged, and some > journals that are clearly mis-categorized, as mentioned in my > recent Scientometrics [v64(3), p351-374] paper with Klavans > and Borner. We can have lots of discussion about the details, > but when you look at the whole thing, it's a good start. > > Perhaps I'm more comfortable than most with the ISI structure > because I don't rely on it for evaluation. We use the > emergent cluster structure from papers rather than any > category or journal scheme. But that's another story. > > Can anyone recommend a better journal categorization scheme > that is widely available? I'd love to hear of (and get a copy > of) such a thing. > > Best regards, > Kevin > > > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Loet Leydesdorff > Sent: Saturday, March 04, 2006 6:15 AM > To: SIGMETRICS at listserv.utk.edu > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Dear colleagues, > > The idea is interesting. However, there a few problems with > this paper. > First, the authors should not have used Pearson correlation > coefficients to compare the rankings, but rank correlations > (Spearman's rho or Kendall's tau). Second, it would have been > interesting to have a rank correlation with "total cites" > given recent discussions (Bensman). > Third, the delineation of fields in terms of the ISI subject > categories is very questionnable. > > However, the authors are very clear about their results: "We > identified ... > , but were unable to recognize a meaningful pattern in the > results." (p. > 9). > I don't understand why one should then multiply the one > measure with the other. What does multiplication to the error? > > Does one of you know a place where the ISI subject categories > are justified? > How are they produced? People seem to use them increasingly > both in evaluation and research practices, but I have never > been able to reproduce them using journal citation measures. > > With best wishes, > > > Loet > > > ________________________________ > Loet Leydesdorff > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal 48, 1012 CX Amsterdam. > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; http://www.leydesdorff.net/ > > > > > -----Original Message----- > > From: ASIS&T Special Interest Group on Metrics > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Eugene Garfield > > Sent: Friday, March 03, 2006 6:37 PM > > To: SIGMETRICS at LISTSERV.UTK.EDU > > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > Further to yesterday's posting, "Prestige is factored into journal > > ratings", here is another interesting and informative article > > > > FULL TEXT AVAILABLE AT : > > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf > > > > email: {jbollen, marko, herbertv}@lanl.gov > > > > TITLE : Journal Status > > > > AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert > Van de Sompel > > > > SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > Abstract > > The status of an actor in a social context is commonly defined in > > terms of two factors: the total number of endorsements the actor > > receives from other actors and the prestige of the endorsing actors. > > These two factors indicate the distinction between popularity and > > expert appreciation of the actor, respectively. We refer to > the former > > > as popularity and to the latter as prestige. These notions of > > popularity and prestige also apply to the domain of scholarly > > assessment. The ISI Impact Factor (ISI IF) is defined as the mean > > number of citations a journal receives over a 2 year > period. By merely > > > counting the amount of citations and disregarding the > prestige of the > > citing journals, the ISI IF is a metric of popularity, not of > > prestige. We demonstrate how a weighted version of the popular > > PageRank algorithm can be used to obtain a metric that reflects > > prestige. We contrast the rankings of journals according to > their ISI > > IF and their weighted PageRank, and we provide an analysis that > > reveals both significant overlaps and differences. > > Furthermore, we introduce the Y-factor which is a simple > combination > > of both the ISI IF and the weighted PageRank, and find that the > > resulting journal rankings correspond well to a general > understanding > > of journal status. > > > > > > ______________________________________________ > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > FULL TEXT AVAILABLE AT : > > > http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR > > http://guide.labanimal.com/news/2006/060213/full/439770a.html > > > > > > Philip Ball : p.ball at nature.com > > www.philipball.com > > > > Title: Prestige is factored into journal ratings > > > > Author(s): Ball P > > > > Source: NATURE 439 (7078): 770-771 FEB 16 2006 > > > > Document Type: News Item Language: English > > Cited References: 0 Times Cited: 0 > > > > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 > CRINAN ST, > > LONDON > > N1 9XW, ENGLAND > > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA > > > > ISSN: 0028-0836 > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fig12.jpg Type: image/jpeg Size: 36597 bytes Desc: not available URL: From loet at LEYDESDORFF.NET Tue Mar 7 03:00:30 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Tue, 7 Mar 2006 09:00:30 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: Thanks, Stephen: I find this very clear. But I thought that you had not found such a high correlation between IFs and total cites. When I compute this correlation over the SCI 2004, I find r = .405 and Spearman's rho = .726. Not too bad! (For the SoSCI r = 0.642 and rho = .746. Of course, everything is highly significant because of the high numbers.) Nevertheless, the picture which I sent you yesterday clearly shows that there are two factors involved: total cites are like total number of publications and faculty assessment based on properties of the journal, that is, the nodes of the networks, while IF's and c/p ratios are properties (e.g., averages) of links. Network parameters are different from node attributes. Both are interesting, but different. With kind regards, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Stephen J Bensman > Sent: Monday, March 06, 2006 11:35 PM > To: SIGMETRICS at listserv.utk.edu > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > Loet, > You and I are perhaps looking at two different aspects of the > problem. I am looking at it from the perspective of a > librarian trying to decide which journals should be provided > with permanent access on a subscription basis and which > should be accessed through some form of intermittent document > delivery. Therefore, I am interested not only in prestige > but also in functionality, i.e., what function does the > journal serve--reporting of research, reviewing of > literature, assistance in teaching, provision of current > news, etc. Citation measures either capture one facet of > functionality--total citations seem to capture reporting of > research, impact factor, review literature--or fail to > capture the functionality at all, i.e., teaching or reporting > of current news. Total citations cannot capture the review > literature, because review journals are usually very small > even though highly rated by scientists, but since impact > factor captures both review literature and current research > significance--which is usually the same as historical > historical research significance due the stability of > patterns--and the correlation of total citations with impact > factor is high enough so that journals high on both can be > captured in a broad category robust against random error, it > seems to me that impact factor can capture two facets of > functionality unlike total citations, which can only capture > one. The hypothesis remains to be tested. > > Prestige appears to operate separately from functionality. > The greatest cause of variance in all four measures is their > belonging to the category of US association journals. The > journals of the American Chemical Society are dominant on all > four measures. Through various evaluations of US > research-doctorate programs by peer ratings and citations, I > can trace this dominance to scientists employed by the > traditionally elite US research insitutions. Thus, variance > and prestige in all four measures is a function of the social > stratification system of US scientific institutions. > There remains the question of how do foreign scientists > relate to the US social stratification system. If they form > a part of it, the foreigners can use ISI citations for > evaluation and other puposes. If they do not form part of > it, then foreigners using ISI citations may only be rating > themselves by how much their work is being accepted by the > scientists within this system. I have no answer to this question. > > I suppose that now you are more confused than ever. I did my > best, but it is complicated as all hell. > > SB > > > > > Loet Leydesdorff @listserv.utk.edu> on > 03/06/2006 > 03:12:54 PM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at listserv.utk.edu > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > Rodriguez, and Herbert > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > > Dear Stephen, > > I apologize if I dragged you into the discussion with > improper argument, but I did not want to mention your idea of > using total cites as an indicator without providing a proper > reference. > > The reasoning in your posting is difficult for me to follow, > but I look forward to reading the full paper. My experience > is that reading the full paper, one begins to understand. I > found your previous argument about using total cites very > convincing because of its high correlation with faculty > ratings and its orthogonality to the impact factor. It seemed > to me that the impact factor measures something very > different from the prestige of a journal. > (Embedded image moved to file: pic17086.gif) Figure 1: > Component plot in rotated space (sources: JCR, 1993; Bensman, > 2001; forthcoming; Bensman & Wilder, 1998). > > > From: Visualization of the Citation Impact Environments of Scientific > Journals: An online mapping exercise, Journal of the American > Society for Information Science and Technology (forthcoming). > . > > > With best wishes, > > > Loet > > ________________________________ > Loet Leydesdorff > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal 48, 1012 CX Amsterdam. > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; http://www.leydesdorff.net/ > > > > > -----Original Message----- > > From: ASIS&T Special Interest Group on Metrics > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman > > Sent: Monday, March 06, 2006 6:41 PM > > To: SIGMETRICS at LISTSERV.UTK.EDU > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and > > Herbert Van de Sompel "Journal Status" > > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > > > > Loet, > > I see that you have once again taken my name in vain and > again given > > me the opportunity to spout my ideas on SIGMETRICS. I must > admit that > > I have not read the paper you discuss, because my doctor warned me > > against reading too many such papers, since I am fairly close to > > OD-ing on them. > > However, the conclusions you mention do seem a little peculiar. > > > > Due to detailed study of Gene Garfield's development and > utilization > > of impact factor, I am coming to change my mind on this measure > > somewhat. It is for rather complicated reasons, which I > shall try to > > explain below. > > > > In general I think that there is too much random error in citation > > data for the utilization of such precise techniques as > > correlation--Pearson, Spearman, whatever. Much results > from exogenous > > citations due to an inability to define precise sets--a logical > > consequence of Bradford's Law of Scattering and Garfield's Law of > > Concentration. Impact factor suffers from a further source > of error > > due to an inability to classify precisely sources into citable and > > non-citable--something which honest persons can disagree on. > > This inability severely affects the denominator of the > impact factor > > equation. What is therefore needed is a technique that is > crude and > > robust against such error. I have personally found it in the > > chi-square test of independence, which allows the conversion of > > citation measures into ordinal variables defined by broad > categories. > > It also allows one to define the amount of error one is willing to > > accept, i.e., upper 10% vs. > > upper 25%. > > > > Use of this chi-square test may vindicate impact factor by > > demonstrating that it has the same strong relationship to expert > > ratings as do total citations. As a matter of fact, it may be a > > superior measure in that it will not only capture the importance of > > reseach journals but also of review journals. Close > inspection of the > > top 10% of the journals recommended by the LSU chemistry faculty > > reveals it to be a balanced mix of research journals, review, > > journals, and the main teaching journal of chemistry. > > In other words, most facets of journal importance are > captured by this > > measure, whereas total citations captures mainly research, > and impact > > factor captures chiefly the review journals. However, > broadening the > > categories may cause impact factor to capture both research > and review > > though not the teaching facet. In any case I am going to > test this in > > the revision of the JASIST paper I am now engaged in. > > > > Impact factor has the ability to do this for the very > reasons Seglen > > denounces it. His main case against is based on the > reasoning of the > > law of error and the role of the arithmetic mean in this law. This > > requires the normal distribution for the arithmetic mean to be an > > accurate estimate of central tendency. However, due to the highly > > skewed distributions with which we deal, the arithmetic > mean is always > > way above the other estimates of central tendency such as > the median > > or the geometric mean due to the high degree of variance caused the > > dominant observations. Seglen's reasoning collapses once > one realizes > > that a journal's or scientist's importance is not measured > by central > > tendency but by the variance caused by the few important articles > > published by the journal or scientist. > > Therefore, scientific importance is the result of variance and not > > central tendency. The arithmetic mean, which impact factor > attempts > > to estimate, better captures the variance. > > > > To demonstrate, I have converted Garfield's constant for > the year 1993 > > into binomial p and the Poisson lambda The way I did this > is in the > > attached Excel file. You will see the binomial p is a > lousy 0.0003, > > which converts into a Poisson lambda or Garfield's constant of 2.15 > > for the year. This is the probability or the rate articles > were cited > > in 1993 on the assumption of probabilistic homogeneity. However, > > since there is probabilistic heterogeneity, most articles > have to have > > a citation rate below Garfield's constant. True to form, > of the 5000 > > journals covered that year, 4500 journals were below to Garfield's > > constant. 2.15 is an awful small range to squeeze 4500 > journals into > > and expect meaningful quantitative distinctions. Utilization of a > > central tendency measure puts one right smack in the middle of that > > tight range. Small as this may be, the probabilities and > lambda were > > actually much smaller, for Garfield's constant is based on > the set of > > articles actually cited that year, i.e., it it truncated on > the left > > and does not take into account the articles that could have > been cited > > but were not. I do not have the technical or intellectual > ability to > > estimate this zero class. I do know that Sir Maurice > Kendall backed > > off from the problem when he confronted it in Bradford's > Law, and who > > the hell am I compared to Maurice Kendall. I wish that > somebody would > > write an article understandable to simpletons on how to make such > > estimates. From my perspective, this would be one of the most > > important articles ever written. > > > > Sorry for the tirade, but I thought I'd float a few trial > balloons to > > be shot down. > > > > SB > > > > (See attached file: GarConst.xls) > > > > > > > > > > > > > > > > > > Loet Leydesdorff @LISTSERV.UTK.EDU> on > > 03/04/2006 > > 07:14:57 AM > > > > Please respond to ASIS&T Special Interest Group on Metrics > > > > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > > Rodriguez, and Herbert > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > Dear colleagues, > > > > The idea is interesting. However, there a few problems with this > > paper. > > First, the authors should not have used Pearson correlation > > coefficients to compare the rankings, but rank correlations > > (Spearman's rho or Kendall's tau). Second, it would have been > > interesting to have a rank correlation with "total cites" > > given recent discussions (Bensman). Third, the delineation > of fields > > in terms of the ISI subject categories is very questionnable. > > > > However, the authors are very clear about their results: "We > > identified ... > > , but were unable to recognize a meaningful pattern in the > results." > > (p. > > 9). > > I don't understand why one should then multiply the one > measure with > > the other. What does multiplication to the error? > > > > Does one of you know a place where the ISI subject categories are > > justified? > > How are they produced? People seem to use them increasingly both in > > evaluation and research practices, but I have never been able to > > reproduce them using journal citation measures. > > > > With best wishes, > > > > > > Loet > > > > > > ________________________________ > > Loet Leydesdorff > > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal > > 48, 1012 CX Amsterdam. > > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; > > http://www.leydesdorff.net/ > > > > > > > > > -----Original Message----- > > > From: ASIS&T Special Interest Group on Metrics > > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Eugene Garfield > > > Sent: Friday, March 03, 2006 6:37 PM > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert > > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > > 9 Jan 2006 > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > Further to yesterday's posting, "Prestige is factored > into journal > > > ratings", here is another interesting and informative article > > > > > > FULL TEXT AVAILABLE AT : > > > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf > > > > > > email: {jbollen, marko, herbertv}@lanl.gov > > > > > > TITLE : Journal Status > > > > > > AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert > > Van de Sompel > > > > > > SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > > > Abstract > > > The status of an actor in a social context is commonly defined in > > > terms of two factors: the total number of endorsements the actor > > > receives from other actors and the prestige of the > > endorsing actors. > > > These two factors indicate the distinction between popularity and > > > expert appreciation of the actor, respectively. We refer to > > the former > > > as popularity and to the latter as prestige. These notions of > > > popularity and prestige also apply to the domain of scholarly > > > assessment. The ISI Impact Factor (ISI IF) is defined as the mean > > > number of citations a journal receives over a 2 year > > period. By merely > > > counting the amount of citations and disregarding the > > prestige of the > > > citing journals, the ISI IF is a metric of popularity, not of > > > prestige. We demonstrate how a weighted version of the popular > > > PageRank algorithm can be used to obtain a metric that reflects > > > prestige. We contrast the rankings of journals according to > > their ISI > > > IF and their weighted PageRank, and we provide an analysis that > > > reveals both significant overlaps and differences. > > > Furthermore, we introduce the Y-factor which is a simple > > combination > > > of both the ISI IF and the weighted PageRank, and find that the > > > resulting journal rankings correspond well to a general > > understanding > > > of journal status. > > > > > > > > > ______________________________________________ > > > > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > FULL TEXT AVAILABLE AT : > > > > > > http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR > > > http://guide.labanimal.com/news/2006/060213/full/439770a.html > > > > > > > > > Philip Ball : p.ball at nature.com > > > www.philipball.com > > > > > > Title: Prestige is factored into journal ratings > > > > > > Author(s): Ball P > > > > > > Source: NATURE 439 (7078): 770-771 FEB 16 2006 > > > > > > Document Type: News Item Language: English > > > Cited References: 0 Times Cited: 0 > > > > > > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 > > CRINAN ST, > > > LONDON > > > N1 9XW, ENGLAND > > > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA > > > > > > ISSN: 0028-0836 > > > > > > > > From notsjb at LSU.EDU Tue Mar 7 11:40:14 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Tue, 7 Mar 2006 10:40:14 -0600 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: Loet, I looked at your picture and will try to interpret it in my terms. I see that LSU faculty score, UIUC use, and total citations are clustered tightly together, whereas impact factor is isolated and apart. Scientist ratings, library use, and total citations are heavily influenced by size. For example, when scientist ratings of the quality of chemistry programs are correlated with the average citations per faculty, the correlation is 0.81, but when these same ratings are correlated with total program citations, the correlation rises to 0.91. In his writings Garfield often commented upon the relationship of size to signifcance, calling upon national journals to consolidate into regional journals to gain added heft. I notice that lately there has been a move among European associations to consolidate their national journals into European ones. Library use and total citations are size dependent in two senses--physical size of journals and length of backfile. Impact factor deliberately controls for both components of size. Moreover, problems with classifying sources into citable and non-citable causes it to have more random error, and this random error operates within a tightly constricted range, causing greater scatter in plots. However, here, also, the distributions are so exponentially skewed, that it readily identifies those journals, whose articles have a higher probability of being cited for reasons of function, prestige, or both. The correlations of total citations with impact factor particularly after the removal of review journal outliers are so high that a cruder method measuring association within broad categories that is more robust against error may reveal a high degree of overlap at the top for all four measures. I have to test for this. Does this clarify matters for you? SB Loet Leydesdorff @LISTSERV.UTK.EDU> on 03/07/2006 02:00:30 AM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Thanks, Stephen: I find this very clear. But I thought that you had not found such a high correlation between IFs and total cites. When I compute this correlation over the SCI 2004, I find r = .405 and Spearman's rho = .726. Not too bad! (For the SoSCI r = 0.642 and rho = .746. Of course, everything is highly significant because of the high numbers.) Nevertheless, the picture which I sent you yesterday clearly shows that there are two factors involved: total cites are like total number of publications and faculty assessment based on properties of the journal, that is, the nodes of the networks, while IF's and c/p ratios are properties (e.g., averages) of links. Network parameters are different from node attributes. Both are interesting, but different. With kind regards, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Stephen J Bensman > Sent: Monday, March 06, 2006 11:35 PM > To: SIGMETRICS at listserv.utk.edu > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > Loet, > You and I are perhaps looking at two different aspects of the > problem. I am looking at it from the perspective of a > librarian trying to decide which journals should be provided > with permanent access on a subscription basis and which > should be accessed through some form of intermittent document > delivery. Therefore, I am interested not only in prestige > but also in functionality, i.e., what function does the > journal serve--reporting of research, reviewing of > literature, assistance in teaching, provision of current > news, etc. Citation measures either capture one facet of > functionality--total citations seem to capture reporting of > research, impact factor, review literature--or fail to > capture the functionality at all, i.e., teaching or reporting > of current news. Total citations cannot capture the review > literature, because review journals are usually very small > even though highly rated by scientists, but since impact > factor captures both review literature and current research > significance--which is usually the same as historical > historical research significance due the stability of > patterns--and the correlation of total citations with impact > factor is high enough so that journals high on both can be > captured in a broad category robust against random error, it > seems to me that impact factor can capture two facets of > functionality unlike total citations, which can only capture > one. The hypothesis remains to be tested. > > Prestige appears to operate separately from functionality. > The greatest cause of variance in all four measures is their > belonging to the category of US association journals. The > journals of the American Chemical Society are dominant on all > four measures. Through various evaluations of US > research-doctorate programs by peer ratings and citations, I > can trace this dominance to scientists employed by the > traditionally elite US research insitutions. Thus, variance > and prestige in all four measures is a function of the social > stratification system of US scientific institutions. > There remains the question of how do foreign scientists > relate to the US social stratification system. If they form > a part of it, the foreigners can use ISI citations for > evaluation and other puposes. If they do not form part of > it, then foreigners using ISI citations may only be rating > themselves by how much their work is being accepted by the > scientists within this system. I have no answer to this question. > > I suppose that now you are more confused than ever. I did my > best, but it is complicated as all hell. > > SB > > > > > Loet Leydesdorff @listserv.utk.edu> on > 03/06/2006 > 03:12:54 PM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at listserv.utk.edu > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > Rodriguez, and Herbert > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > > Dear Stephen, > > I apologize if I dragged you into the discussion with > improper argument, but I did not want to mention your idea of > using total cites as an indicator without providing a proper > reference. > > The reasoning in your posting is difficult for me to follow, > but I look forward to reading the full paper. My experience > is that reading the full paper, one begins to understand. I > found your previous argument about using total cites very > convincing because of its high correlation with faculty > ratings and its orthogonality to the impact factor. It seemed > to me that the impact factor measures something very > different from the prestige of a journal. > (Embedded image moved to file: pic17086.gif) Figure 1: > Component plot in rotated space (sources: JCR, 1993; Bensman, > 2001; forthcoming; Bensman & Wilder, 1998). > > > From: Visualization of the Citation Impact Environments of Scientific > Journals: An online mapping exercise, Journal of the American > Society for Information Science and Technology (forthcoming). > . > > > With best wishes, > > > Loet > > ________________________________ > Loet Leydesdorff > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal 48, 1012 CX Amsterdam. > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; http://www.leydesdorff.net/ > > > > > -----Original Message----- > > From: ASIS&T Special Interest Group on Metrics > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman > > Sent: Monday, March 06, 2006 6:41 PM > > To: SIGMETRICS at LISTSERV.UTK.EDU > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and > > Herbert Van de Sompel "Journal Status" > > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > > > > Loet, > > I see that you have once again taken my name in vain and > again given > > me the opportunity to spout my ideas on SIGMETRICS. I must > admit that > > I have not read the paper you discuss, because my doctor warned me > > against reading too many such papers, since I am fairly close to > > OD-ing on them. > > However, the conclusions you mention do seem a little peculiar. > > > > Due to detailed study of Gene Garfield's development and > utilization > > of impact factor, I am coming to change my mind on this measure > > somewhat. It is for rather complicated reasons, which I > shall try to > > explain below. > > > > In general I think that there is too much random error in citation > > data for the utilization of such precise techniques as > > correlation--Pearson, Spearman, whatever. Much results > from exogenous > > citations due to an inability to define precise sets--a logical > > consequence of Bradford's Law of Scattering and Garfield's Law of > > Concentration. Impact factor suffers from a further source > of error > > due to an inability to classify precisely sources into citable and > > non-citable--something which honest persons can disagree on. > > This inability severely affects the denominator of the > impact factor > > equation. What is therefore needed is a technique that is > crude and > > robust against such error. I have personally found it in the > > chi-square test of independence, which allows the conversion of > > citation measures into ordinal variables defined by broad > categories. > > It also allows one to define the amount of error one is willing to > > accept, i.e., upper 10% vs. > > upper 25%. > > > > Use of this chi-square test may vindicate impact factor by > > demonstrating that it has the same strong relationship to expert > > ratings as do total citations. As a matter of fact, it may be a > > superior measure in that it will not only capture the importance of > > reseach journals but also of review journals. Close > inspection of the > > top 10% of the journals recommended by the LSU chemistry faculty > > reveals it to be a balanced mix of research journals, review, > > journals, and the main teaching journal of chemistry. > > In other words, most facets of journal importance are > captured by this > > measure, whereas total citations captures mainly research, > and impact > > factor captures chiefly the review journals. However, > broadening the > > categories may cause impact factor to capture both research > and review > > though not the teaching facet. In any case I am going to > test this in > > the revision of the JASIST paper I am now engaged in. > > > > Impact factor has the ability to do this for the very > reasons Seglen > > denounces it. His main case against is based on the > reasoning of the > > law of error and the role of the arithmetic mean in this law. This > > requires the normal distribution for the arithmetic mean to be an > > accurate estimate of central tendency. However, due to the highly > > skewed distributions with which we deal, the arithmetic > mean is always > > way above the other estimates of central tendency such as > the median > > or the geometric mean due to the high degree of variance caused the > > dominant observations. Seglen's reasoning collapses once > one realizes > > that a journal's or scientist's importance is not measured > by central > > tendency but by the variance caused by the few important articles > > published by the journal or scientist. > > Therefore, scientific importance is the result of variance and not > > central tendency. The arithmetic mean, which impact factor > attempts > > to estimate, better captures the variance. > > > > To demonstrate, I have converted Garfield's constant for > the year 1993 > > into binomial p and the Poisson lambda The way I did this > is in the > > attached Excel file. You will see the binomial p is a > lousy 0.0003, > > which converts into a Poisson lambda or Garfield's constant of 2.15 > > for the year. This is the probability or the rate articles > were cited > > in 1993 on the assumption of probabilistic homogeneity. However, > > since there is probabilistic heterogeneity, most articles > have to have > > a citation rate below Garfield's constant. True to form, > of the 5000 > > journals covered that year, 4500 journals were below to Garfield's > > colstant. 2.15 is an awful small range to squeeze 4500 > journals into > > and expect meaningful quantitative distinctions. Utilization of a > > central tendency measure puts one right smack in the middle of that > > tight range. Small as this may be, the probabilities and > lambda were > > actually much smaller, for Garfield's constant is based on > the set of > > articles actually cited that year, i.e., it it truncated on > the left > > and does not take into account the articles that could have > been cited > > but were not. I do not have the technical or intellectual > ability to > > estimate this zero class. I do know that Sir Maurice > Kendall backed > > off from the problem when he confronted it in Bradford's > Law, and who > > the hell am I compared to Maurice Kendall. I wish that > somebody would > > write an article understandable to simpletons on how to make such > > estimates. From my perspective, this would be one of the most > > important articles ever written. > > > > Sorry for the tirade, but I thought I'd float a few trial > balloons to > > be shot down. > > > > SB > > > > (See attached file: GarConst.xls) > > > > > > > > > > > > > > > > > > Loet Leydesdorff @LISTSERV.UTK.EDU> on > > 03/04/2006 > > 07:14:57 AM > > > > Please respond to ASIS&T Special Interest Group on Metrics > > > > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > > Rodriguez, and Herbert > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > Dear colleagues, > > > > The idea is interesting. However, there a few problems with this > > paper. > > First, the authors should not have used Pearson correlation > > coefficients to compare the rankings, but rank correlations > > (Spearman's rho or Kendall's tau). Second, it would have been > > interesting to have a rank correlation with "total cites" > > given recent discussions (Bensman). Third, the delineation > of fields > > in terms of the ISI subject categories is very questionnable. > > > > However, the authors are very clear about their results: "We > > identified ... > > , but were unable to recognize a meaningful pattern in the > results." > > (p. > > 9). > > I don't understand why one should then multiply the one > measure with > > the other. What does multiplication to the error? > > > > Does one of you know a place where the ISI subject categories are > > justified? > > How are they produced? People seem to use them increasingly both in > > evaluation and research practices, but I have never been able to > > reproduce them using journal citation measures. > > > > With best wishes, > > > > > > Loet > > > > > > ________________________________ > > Loet Leydesdorff > > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal > > 48, 1012 CX Amsterdam. > > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; > > http://www.leydesdorff.net/ > > > > > > > > > -----Original Message----- > > > From: ASIS&T Special Interest Group on Metrics > > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Eugene Garfield > > > Sent: Friday, March 03, 2006 6:37 PM > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert > > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > > 9 Jan 2006 > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > Further to yesterday's posting, "Prestige is factored > into journal > > > ratings", here is another interesting and informative article > > > > > > FULL TEXT AVAILABLE AT : > > > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf > > > > > > email: {jbollen, marko, herbertv}@lanl.gov > > > > > > TITLE : Journal Status > > > > > > AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert > > Van de Sompel > > > > > > SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > > > Abstract > > > The status of an actor in a social context is commonly defined in > > > terms of two factors: the total number of endorsements the actor > > > receives from other actors and the prestige of the > > endorsing actors. > > > These two factors indicate the distinction between popularity and > > > expert appreciation of the actor, respectively. We refer to > > the former > > > as popularity and to the latter as prestige. These notions of > > > popularity and prestige also apply to the domain of scholarly > > > assessment. The ISI Impact Factor (ISI IF) is defined as the mean > > > number of citations a journal receives over a 2 year > > period. By merely > > > counting the amount of citations and disregarding the > > prestige of the > > > citing journals, the ISI IF is a metric of popularity, not of > > > prestige. We demonstrate how a weighted version of the popular > > > PageRank algorithm can be used to obtain a metric that reflects > > > prestige. We contrast the rankings of journals according to > > their ISI > > > IF and their weighted PageRank, and we provide an analysis that > > > reveals both significant overlaps and differences. > > > Furthermore, we introduce the Y-factor which is a simple > > combination > > > of both the ISI IF and the weighted PageRank, and find that the > > > resulting journal rankings correspond well to a general > > understanding > > > of journal status. > > > > > > > > > ______________________________________________ > > > > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > FULL TEXT AVAILABLE AT : > > > > > > http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR > > > http://guide.labanimal.com/news/2006/060213/full/439770a.html > > > > > > > > > Philip Ball : p.ball at nature.com > > > www.philipball.com > > > > > > Title: Prestige is factored into journal ratings > > > > > > Author(s): Ball P > > > > > > Source: NATURE 439 (7078): 770-771 FEB 16 2006 > > > > > > Document Type: News Item Language: English > > > Cited References: 0 Times Cited: 0 > > > > > > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 > > CRINAN ST, > > > LONDON > > > N1 9XW, ENGLAND > > > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA > > > > > > ISSN: 0028-0836 > > > > > > > > From loet at LEYDESDORFF.NET Tue Mar 7 12:38:56 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Tue, 7 Mar 2006 18:38:56 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: Dear Stephen, It seems that we agree that there is a strong factor correlating size, number of citations, number of publications, faculty appreciation, etc., which is orthogonal to the impact factor (when using your dataset). Within your dataset, you can make these statements about the quality of the impact factor because it is relatively a disciplinarily homogenous set of chemistry journals. This would break down in the case of more mixed sets. I can do it for the complete set, but I don't have faculty appreciations; only total citations, nr of publications and impact factors. The expectation is that I find the two factors again because impact factors are not really different from c/p ratios (with a delay), and c/p ratios are the quotients of two of the indicators on the other axis. c/p ratios can be expected to correlate highly with IFs, but not with c or p themselves. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman > Sent: Tuesday, March 07, 2006 5:40 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Loet, > I looked at your picture and will try to interpret it in my > terms. I see that LSU faculty score, UIUC use, and total > citations are clustered tightly together, whereas impact > factor is isolated and apart. Scientist ratings, library > use, and total citations are heavily influenced by size. For > example, when scientist ratings of the quality of chemistry > programs are correlated with the average citations per > faculty, the correlation is 0.81, but when these same ratings > are correlated with total program citations, the correlation > rises to 0.91. In his writings Garfield often commented upon > the relationship of size to signifcance, calling upon > national journals to consolidate into regional journals to > gain added heft. I notice that lately there has been a move > among European associations to consolidate their national > journals into European ones. > > Library use and total citations are size dependent in two > senses--physical size of journals and length of backfile. > > Impact factor deliberately controls for both components of > size. Moreover, problems with classifying sources into > citable and non-citable causes it to have more random error, > and this random error operates within a tightly constricted > range, causing greater scatter in plots. However, here, > also, the distributions are so exponentially skewed, that it > readily identifies those journals, whose articles have a > higher probability of being cited for reasons of function, > prestige, or both. The correlations of total citations with > impact factor particularly after the removal of review > journal outliers are so high that a cruder method measuring > association within broad categories that is more robust > against error may reveal a high degree of overlap at the top > for all four measures. I have to test for this. > > Does this clarify matters for you? > > SB > > > > > > > Loet Leydesdorff @LISTSERV.UTK.EDU> on > 03/07/2006 02:00:30 AM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > Rodriguez, and Herbert > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Thanks, Stephen: I find this very clear. > > But I thought that you had not found such a high correlation > between IFs and total cites. When I compute this correlation > over the SCI 2004, I find r = > .405 and Spearman's rho = .726. Not too bad! (For the SoSCI r > = 0.642 and rho = .746. Of course, everything is highly > significant because of the high > numbers.) > > Nevertheless, the picture which I sent you yesterday clearly > shows that there are two factors involved: total cites are > like total number of publications and faculty assessment > based on properties of the journal, that is, the nodes of the > networks, while IF's and c/p ratios are properties (e.g., > averages) of links. > > Network parameters are different from node attributes. Both > are interesting, but different. > > With kind regards, > > > Loet > > ________________________________ > Loet Leydesdorff > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal 48, 1012 CX Amsterdam. > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; http://www.leydesdorff.net/ > > > > > -----Original Message----- > > From: ASIS&T Special Interest Group on Metrics > > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Stephen J Bensman > > Sent: Monday, March 06, 2006 11:35 PM > > To: SIGMETRICS at listserv.utk.edu > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and > > Herbert Van de Sompel "Journal Status" > > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > > > > Loet, > > You and I are perhaps looking at two different aspects of > the problem. > > I am looking at it from the perspective of a librarian trying to > > decide which journals should be provided with permanent access on a > > subscription basis and which should be accessed through > some form of > > intermittent document delivery. Therefore, I am interested > not only > > in prestige but also in functionality, i.e., what function does the > > journal serve--reporting of research, reviewing of literature, > > assistance in teaching, provision of current news, etc. Citation > > measures either capture one facet of functionality--total citations > > seem to capture reporting of research, impact factor, review > > literature--or fail to capture the functionality at all, i.e., > > teaching or reporting of current news. Total citations > cannot capture > > the review literature, because review journals are usually > very small > > even though highly rated by scientists, but since impact factor > > captures both review literature and current research > > significance--which is usually the same as historical historical > > research significance due the stability of patterns--and the > > correlation of total citations with impact factor is high enough so > > that journals high on both can be captured in a broad > category robust > > against random error, it seems to me that impact factor can capture > > two facets of functionality unlike total citations, which can only > > capture one. The hypothesis remains to be tested. > > > > Prestige appears to operate separately from functionality. > > The greatest cause of variance in all four measures is > their belonging > > to the category of US association journals. The journals of the > > American Chemical Society are dominant on all four > measures. Through > > various evaluations of US research-doctorate programs by > peer ratings > > and citations, I can trace this dominance to scientists employed by > > the traditionally elite US research insitutions. Thus, > variance and > > prestige in all four measures is a function of the social > > stratification system of US scientific institutions. > > There remains the question of how do foreign scientists > relate to the > > US social stratification system. If they form a part of it, the > > foreigners can use ISI citations for evaluation and other > puposes. If > > they do not form part of it, then foreigners using ISI > citations may > > only be rating themselves by how much their work is being > accepted by > > the scientists within this system. I have no answer to > this question. > > > > I suppose that now you are more confused than ever. I did my best, > > but it is complicated as all hell. > > > > SB > > > > > > > > > > Loet Leydesdorff @listserv.utk.edu> on > > 03/06/2006 > > 03:12:54 PM > > > > Please respond to ASIS&T Special Interest Group on Metrics > > > > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > > > > > To: SIGMETRICS at listserv.utk.edu > > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > > Rodriguez, and Herbert > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > Dear Stephen, > > > > I apologize if I dragged you into the discussion with improper > > argument, but I did not want to mention your idea of using > total cites > > as an indicator without providing a proper reference. > > > > The reasoning in your posting is difficult for me to follow, but I > > look forward to reading the full paper. My experience is > that reading > > the full paper, one begins to understand. I found your previous > > argument about using total cites very convincing because of > its high > > correlation with faculty ratings and its orthogonality to > the impact > > factor. It seemed to me that the impact factor measures > something very > > different from the prestige of a journal. > > (Embedded image moved to file: pic17086.gif) Figure 1: > > Component plot in rotated space (sources: JCR, 1993; Bensman, 2001; > > forthcoming; Bensman & Wilder, 1998). > > > > > > From: Visualization of the Citation Impact Environments of > Scientific > > Journals: An online mapping exercise, Journal of the > American Society > > for Information Science and Technology (forthcoming). > > . > > > > > > With best wishes, > > > > > > Loet > > > > ________________________________ > > Loet Leydesdorff > > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal > > 48, 1012 CX Amsterdam. > > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; > > http://www.leydesdorff.net/ > > > > > > > > > -----Original Message----- > > > From: ASIS&T Special Interest Group on Metrics > > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen > J Bensman > > > Sent: Monday, March 06, 2006 6:41 PM > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and > > > Herbert Van de Sompel "Journal Status" > > > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > > > > > > > > > > Loet, > > > I see that you have once again taken my name in vain and > > again given > > > me the opportunity to spout my ideas on SIGMETRICS. I must > > admit that > > > I have not read the paper you discuss, because my doctor > warned me > > > against reading too many such papers, since I am fairly close to > > > OD-ing on them. > > > However, the conclusions you mention do seem a little peculiar. > > > > > > Due to detailed study of Gene Garfield's development and > > utilization > > > of impact factor, I am coming to change my mind on this measure > > > somewhat. It is for rather complicated reasons, which I > > shall try to > > > explain below. > > > > > > In general I think that there is too much random error in > citation > > > data for the utilization of such precise techniques as > > > correlation--Pearson, Spearman, whatever. Much results > > from exogenous > > > citations due to an inability to define precise sets--a logical > > > consequence of Bradford's Law of Scattering and Garfield's Law of > > > Concentration. Impact factor suffers from a further source > > of error > > > due to an inability to classify precisely sources into > citable and > > > non-citable--something which honest persons can disagree on. > > > This inability severely affects the denominator of the > > impact factor > > > equation. What is therefore needed is a technique that is > > crude and > > > robust against such error. I have personally found it in the > > > chi-square test of independence, which allows the conversion of > > > citation measures into ordinal variables defined by broad > > categories. > > > It also allows one to define the amount of error one is > willing to > > > accept, i.e., upper 10% vs. > > > upper 25%. > > > > > > Use of this chi-square test may vindicate impact factor by > > > demonstrating that it has the same strong relationship to expert > > > ratings as do total citations. As a matter of fact, it may be a > > > superior measure in that it will not only capture the > importance of > > > reseach journals but also of review journals. Close > > inspection of the > > > top 10% of the journals recommended by the LSU chemistry faculty > > > reveals it to be a balanced mix of research journals, review, > > > journals, and the main teaching journal of chemistry. > > > In other words, most facets of journal importance are > > captured by this > > > measure, whereas total citations captures mainly research, > > and impact > > > factor captures chiefly the review journals. However, > > broadening the > > > categories may cause impact factor to capture both research > > and review > > > though not the teaching facet. In any case I am going to > > test this in > > > the revision of the JASIST paper I am now engaged in. > > > > > > Impact factor has the ability to do this for the very > > reasons Seglen > > > denounces it. His main case against is based on the > > reasoning of the > > > law of error and the role of the arithmetic mean in this > law. This > > > requires the normal distribution for the arithmetic mean to be an > > > accurate estimate of central tendency. However, due to > the highly > > > skewed distributions with which we deal, the arithmetic > > mean is always > > > way above the other estimates of central tendency such as > > the median > > > or the geometric mean due to the high degree of variance > caused the > > > dominant observations. Seglen's reasoning collapses once > > one realizes > > > that a journal's or scientist's importance is not measured > > by central > > > tendency but by the variance caused by the few important articles > > > published by the journal or scientist. > > > Therefore, scientific importance is the result of > variance and not > > > central tendency. The arithmetic mean, which impact factor > > attempts > > > to estimate, better captures the variance. > > > > > > To demonstrate, I have converted Garfield's constant for > > the year 1993 > > > into binomial p and the Poisson lambda The way I did this > > is in the > > > attached Excel file. You will see the binomial p is a > > lousy 0.0003, > > > which converts into a Poisson lambda or Garfield's > constant of 2.15 > > > for the year. This is the probability or the rate articles > > were cited > > > in 1993 on the assumption of probabilistic homogeneity. However, > > > since there is probabilistic heterogeneity, most articles > > have to have > > > a citation rate below Garfield's constant. True to form, > > of the 5000 > > > journals covered that year, 4500 journals were below to > Garfield's > > > colstant. 2.15 is an awful small range to squeeze 4500 > > journals into > > > and expect meaningful quantitative distinctions. > Utilization of a > > > central tendency measure puts one right smack in the > middle of that > > > tight range. Small as this may be, the probabilities and > > lambda were > > > actually much smaller, for Garfield's constant is based on > > the set of > > > articles actually cited that year, i.e., it it truncated on > > the left > > > and does not take into account the articles that could have > > been cited > > > but were not. I do not have the technical or intellectual > > ability to > > > estimate this zero class. I do know that Sir Maurice > > Kendall backed > > > off from the problem when he confronted it in Bradford's > > Law, and who > > > the hell am I compared to Maurice Kendall. I wish that > > somebody would > > > write an article understandable to simpletons on how to make such > > > estimates. From my perspective, this would be one of the most > > > important articles ever written. > > > > > > Sorry for the tirade, but I thought I'd float a few trial > > balloons to > > > be shot down. > > > > > > SB > > > > > > (See attached file: GarConst.xls) > > > > > > > > > > > > > > > > > > > > > > > > > > > Loet Leydesdorff @LISTSERV.UTK.EDU> on > > > 03/04/2006 > > > 07:14:57 AM > > > > > > Please respond to ASIS&T Special Interest Group on Metrics > > > > > > > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > > > > > > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > > > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > > > Rodriguez, and Herbert > > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > > 9 Jan 2006 > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > Dear colleagues, > > > > > > The idea is interesting. However, there a few problems with this > > > paper. > > > First, the authors should not have used Pearson correlation > > > coefficients to compare the rankings, but rank correlations > > > (Spearman's rho or Kendall's tau). Second, it would have been > > > interesting to have a rank correlation with "total cites" > > > given recent discussions (Bensman). Third, the delineation > > of fields > > > in terms of the ISI subject categories is very questionnable. > > > > > > However, the authors are very clear about their results: "We > > > identified ... > > > , but were unable to recognize a meaningful pattern in the > > results." > > > (p. > > > 9). > > > I don't understand why one should then multiply the one > > measure with > > > the other. What does multiplication to the error? > > > > > > Does one of you know a place where the ISI subject categories are > > > justified? > > > How are they produced? People seem to use them > increasingly both in > > > evaluation and research practices, but I have never been able to > > > reproduce them using journal citation measures. > > > > > > With best wishes, > > > > > > > > > Loet > > > > > > > > > ________________________________ > > > Loet Leydesdorff > > > Amsterdam School of Communications Research (ASCoR), > > Kloveniersburgwal > > > 48, 1012 CX Amsterdam. > > > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > > loet at leydesdorff.net ; > > > http://www.leydesdorff.net/ > > > > > > > > > > > > > -----Original Message----- > > > > From: ASIS&T Special Interest Group on Metrics > > > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of > Eugene Garfield > > > > Sent: Friday, March 03, 2006 6:37 PM > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > > > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > > and Herbert > > > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > > > 9 Jan 2006 > > > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > Further to yesterday's posting, "Prestige is factored > > into journal > > > > ratings", here is another interesting and informative article > > > > > > > > FULL TEXT AVAILABLE AT : > > > > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf > > > > > > > > email: {jbollen, marko, herbertv}@lanl.gov > > > > > > > > TITLE : Journal Status > > > > > > > > AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert > > > Van de Sompel > > > > > > > > SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > > > > > Abstract > > > > The status of an actor in a social context is commonly > defined in > > > > terms of two factors: the total number of endorsements > the actor > > > > receives from other actors and the prestige of the > > > endorsing actors. > > > > These two factors indicate the distinction between > popularity and > > > > expert appreciation of the actor, respectively. We refer to > > > the former > > > > as popularity and to the latter as prestige. These notions of > > > > popularity and prestige also apply to the domain of scholarly > > > > assessment. The ISI Impact Factor (ISI IF) is defined > as the mean > > > > number of citations a journal receives over a 2 year > > > period. By merely > > > > counting the amount of citations and disregarding the > > > prestige of the > > > > citing journals, the ISI IF is a metric of popularity, not of > > > > prestige. We demonstrate how a weighted version of the popular > > > > PageRank algorithm can be used to obtain a metric that reflects > > > > prestige. We contrast the rankings of journals according to > > > their ISI > > > > IF and their weighted PageRank, and we provide an analysis that > > > > reveals both significant overlaps and differences. > > > > Furthermore, we introduce the Y-factor which is a simple > > > combination > > > > of both the ISI IF and the weighted PageRank, and find that the > > > > resulting journal rankings correspond well to a general > > > understanding > > > > of journal status. > > > > > > > > > > > > ______________________________________________ > > > > > > > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > FULL TEXT AVAILABLE AT : > > > > > > > > > > http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR > > > > http://guide.labanimal.com/news/2006/060213/full/439770a.html > > > > > > > > > > > > Philip Ball : p.ball at nature.com > > > > www.philipball.com > > > > > > > > Title: Prestige is factored into journal ratings > > > > > > > > Author(s): Ball P > > > > > > > > Source: NATURE 439 (7078): 770-771 FEB 16 2006 > > > > > > > > Document Type: News Item Language: English > > > > Cited References: 0 Times Cited: 0 > > > > > > > > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 > > > CRINAN ST, > > > > LONDON > > > > N1 9XW, ENGLAND > > > > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA > > > > > > > > ISSN: 0028-0836 > > > > > > > > > > > > > From notsjb at LSU.EDU Tue Mar 7 13:53:58 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Tue, 7 Mar 2006 12:53:58 -0600 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: Loet, Let me offer a suggestion on why impact factor is so separated from the other variables on your graphic. The journals causing most of the variance in impact factor are review journals. Their high average citation results from their function. The other variable, in which review journals are a significant source of variance, is faculty score, which has the two key review journals among its top 12 or 10%, thereby indicating the importance of review journals to scientists. I do notice that faculty score is the closest of the other variables to impact factor on your graphic. Outlier analysis has revealed that review journals have higher library use than warranted by their size--a further indication of their functional importance. My conclusions are thus. Garfield was correct in his employment of impact factor to identify the important review journals among the small journals whose significance would be otherwise inundated by size. The separation of impact factor from the other variables on your graphic is due mainly due to differing function of the journals causing the variance in impact factor. It is not differing prestige, because the key review journals are mostly US association journals like the key research journals. There is a role played by fuzzy sets. The higher citation rates of biochemistry journals forces these journals to the top in impact factor, giving a further reason for the isolation of this variable on your graphic. Most of the biochemistry citations are exogenous to the interests of the LSU chemistry faculty and patrons of the UI Chemistry Library. This is seen from the fact that biochemistry journals rank low on faculty score and library use but high on both the citation measures. How do these explanations play with you? SB Loet Leydesdorff @listserv.utk.edu> on 03/07/2006 11:38:56 AM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at listserv.utk.edu cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Dear Stephen, It seems that we agree that there is a strong factor correlating size, number of citations, number of publications, faculty appreciation, etc., which is orthogonal to the impact factor (when using your dataset). Within your dataset, you can make these statements about the quality of the impact factor because it is relatively a disciplinarily homogenous set of chemistry journals. This would break down in the case of more mixed sets. I can do it for the complete set, but I don't have faculty appreciations; only total citations, nr of publications and impact factors. The expectation is that I find the two factors again because impact factors are not really different from c/p ratios (with a delay), and c/p ratios are the quotients of two of the indicators on the other axis. c/p ratios can be expected to correlate highly with IFs, but not with c or p themselves. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman > Sent: Tuesday, March 07, 2006 5:40 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Loet, > I looked at your picture and will try to interpret it in my > terms. I see that LSU faculty score, UIUC use, and total > citations are clustered tightly together, whereas impact > factor is isolated and apart. Scientist ratings, library > use, and total citations are heavily influenced by size. For > example, when scientist ratings of the quality of chemistry > programs are correlated with the average citations per > faculty, the correlation is 0.81, but when these same ratings > are correlated with total program citations, the correlation > rises to 0.91. In his writings Garfield often commented upon > the relationship of size to signifcance, calling upon > national journals to consolidate into regional journals to > gain added heft. I notice that lately there has been a move > among European associations to consolidate their national > journals into European ones. > > Library use and total citations are size dependent in two > senses--physical size of journals and length of backfile. > > Impact factor deliberately controls for both components of > size. Moreover, problems with classifying sources into > citable and non-citable causes it to have more random error, > and this random error operates within a tightly constricted > range, causing greater scatter in plots. However, here, > also, the distributions are so exponentially skewed, that it > readily identifies those journals, whose articles have a > higher probability of being cited for reasons of function, > prestige, or both. The correlations of total citations with > impact factor particularly after the removal of review > journal outliers are so high that a cruder method measuring > association within broad categories that is more robust > against error may reveal a high degree of overlap at the top > for all four measures. I have to test for this. > > Does this clarify matters for you? > > SB > > > > > > > Loet Leydesdorff @LISTSERV.UTK.EDU> on > 03/07/2006 02:00:30 AM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > Rodriguez, and Herbert > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Thanks, Stephen: I find this very clear. > > But I thought that you had not found such a high correlation > between IFs and total cites. When I compute this correlation > over the SCI 2004, I find r = > .405 and Spearman's rho = .726. Not too bad! (For the SoSCI r > = 0.642 and rho = .746. Of course, everything is highly > significant because of the high > numbers.) > > Nevertheless, the picture which I sent you yesterday clearly > shows that there are two factors involved: total cites are > like total number of publications and faculty assessment > based on properties of the journal, that is, the nodes of the > networks, while IF's and c/p ratios are properties (e.g., > averages) of links. > > Network parameters are different from node attributes. Both > are interesting, but different. > > With kind regards, > > > Loet > > ________________________________ > Loet Leydesdorff > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal 48, 1012 CX Amsterdam. > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; http://www.leydesdorff.net/ > > > > > -----Original Message----- > > From: ASIS&T Special Interest Group on Metrics > > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Stephen J Bensman > > Sent: Monday, March 06, 2006 11:35 PM > > To: SIGMETRICS at listserv.utk.edu > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and > > Herbert Van de Sompel "Journal Status" > > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > > > > Loet, > > You and I are perhaps looking at two different aspects of > the problem. > > I am looking at it from the perspective of a librarian trying to > > decide which journals should be provided with permanent access on a > > subscription basis and which should be accessed through > some form of > > intermittent document delivery. Therefore, I am interested > not only > > in prestige but also in functionality, i.e., what function does the > > journal serve--reporting of research, reviewing of literature, > > assistance in teaching, provision of current news, etc. Citation > > measures either capture one facet of functionality--total citations > > seem to capture reporting of research, impact factor, review > > literature--or fail to capture the functionality at all, i.e., > > teaching or reporting of current news. Total citations > cannot capture > > the review literature, because review journals are usually > very small > > even though highly rated by scientists, but since impact factor > > captures both review literature and current research > > significance--which is usually the same as historical historical > > research significance due the stability of patterns--and the > > correlation of total citations with impact factor is high enough so > > that journals high on both can be captured in a broad > category robust > > against random error, it seems to me that impact factor can capture > > two facets of functionality unlike total citations, which can only > > capture one. The hypothesis remains to be tested. > > > > Prestige appears to operate separately from functionality. > > The greatest cause of variance in all four measures is > their belongine > > to the category of US association journals. The journals mf the > > American Chemical Society are dominant on all four > measures. Through > > various evaluations of US research-doctorate programs by > peer ratings > > and citations, I can trace this dominance to scientists employed by > > the traditionally elite US research insitutions. Thus, > variance and > > prestige in all four measures is a function of the social > > stratification system of US scientific institutions. > > There pemains the question of how do foreign scientists > relate to the > > US social stratification system. If they form a part of it, the > > foreigners can use ISI citations for evaluation and other > puposes. If > > they do not form part of it, then foreigners using ISI > citations may > > only be rating themselves by how much their work is being > accepted by > > the scientists within this system. I have no answer to > this question. > > > > I suppose that now you are more confused than ever. I did my best, > > but it is complicated as all hell. > > > > SB > > > > > > > > > > Loet Leydesdorff @listserv.utk.edu> on > > 03/06/2006 > > 03:12:54 PM > > > > Please respond to ASIS&T Special Interest Group on Metrics > > > > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > > > > > To: SIGMETRICS at listserv.utk.edu > > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > > Rodriguez, and Herbert > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > Dear Stephen, > > > > I apologize if I dragged you into the discussion with improper > > argument, but I did not want to mention your idea of using > total cites > > as an indicator without providing a proper reference. > > > > The reasoning in your posting is difficult for me to follow, but I > > look forward to reading the full paper. My experience is > that reading > > the full paper, one begins to understand. I found your previous > > argument about using total cites very convincing because of > its high > > correlation with faculty ratings and its orthogonality to > the impact > > factor. It seemed to me that the impact factor measures > something very > > different from the prestige of a journal. > > (Embedded image moved to file: pic17086.gif) Figure 1: > > Component plot in rotated space (sources: JCR, 1993; Bensman, 2001; > > forthcoming; Bensman & Wilder, 1998). > > > > > > From: Visualization of the Citation Impact Environments of > Scientific > > Journals: An online mapping exercise, Journal of the > American Society > > for Information Science and Technology (forthcoming). > > . > > > > > > With best wishes, > > > > > > Loet > > > > ________________________________ > > Loet Leydesdorff > > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal > > 48, 1012 CX Amsterdam. > > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; > > http://www.leydesdorff.net/ > > > > > > > > > -----Original Message----- > > > From: ASIS&T Special Interest Group on Metrics > > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen > J Bensman > > > Sent: Monday, March 06, 2006 6:41 PM > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and > > > Herbert Van de Sompel "Journal Status" > > > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > > > > > > > > > > Loet, > > > I see that you have once again taken my name in vain and > > again given > > > me the opportunity to spout my ideas on SIGMETRICS. I must > > admit that > > > I have not read the paper you discuss, because my doctor > warned me > > > against reading too many such papers, since I am fairly close to > > > OD-ing on them. > > > However, the conclusions you mention do seem a little peculiar. > > > > > > Due to detailed study of Gene Garfield's development and > > utilization > > > of impact factor, I am coming to change my mind on this measure > > > somewhat. It is for rather complicated reasons, which I > > shall try to > > > explain below. > > > > > > In general I think that there is too much random error in > citation > > > data for the utilization of such precise techniques as > > > correlation--Pearson, Spearman, whatever. Much results > > from exogenous > > > citations due to an inability to define precise sets--a logical > > > consequence of Bradford's Law of Scattering and Garfield's Law of > > > Concentration. Impact factor suffers from a further source > > of error > > > due to an inability to classify precisely sources into > citable and > > > non-citable--something which honest persons can disagree on. > > > This inability severely affects the denominator of the > > impact factor > > > equation. What is therefore needed is a technique that is > > crude and > > > robust against such error. I have personally found it in the > > > chi-square test of independence, which allows the conversion of > > > citation measures into ordinal variables defined by broad > > categories. > > > It also allows one to define the amount of error one is > willing to > > > accept, i.e., upper 10% vs. > > > upper 25%. > > > > > > Use of this chi-square test may vindicate impact factor by > > > demonstrating that it has the same strong relationship to expert > > > ratings as do total citations. As a matter of fact, it may be a > > > superior measure in that it will not only capture the > importance of > > > reseach journals but also of review journals. Close > > inspection of the > > > top 10% of the journals recommended by the LSU chemistry faculty > > > reveals it to be a balanced mix of research journals, review, > > > journals, and the main teaching journal of chemistry. > > > In other words, most facets of journal importance are > > captured by this > > > measure, whereas total citations captures mainly research, > > and impact > > > factor captures chiefly the review journals. However, > > broadening the > > > categories may cause impact factor to capture both research > > and review > > > though not the teaching facet. In any case I am going to > > test this in > > > the revision of the JASIST paper I am now engaged in. > > > > > > Impact factor has the ability to do this for the very > > reasons Seglen > > > denounces it. His main case against is based on the > > reasoning of the > > > law of error and the role of the arithmetic mean in this > law. This > > > requires the normal distribution for the arithmetic mean to be an > > > accurate estimate of central tendency. However, due to > the highly > > > skewed distributions with which we deal, the arithmetic > > mean is always > > > way above the other estimates of central tendency such as > > the median > > > or the geometric mean due to the high degree of variance > caused the > > > dominant observations. Seglen's reasoning collapses once > > one realizes > > > that a journal's or scientist's importance is not measured > > by central > > > tendency but by the variance caused by the few important articles > > > published by the journal or scientist. > > > Therefore, scientific importance is the result of > variance and not > > > central tendency. The arithmetic mean, which impact factor > > attempts > > > to estimate, better captures the variance. > > > > > > To demonstrate, I have converted Garfield's constant for > > the year 1993 > > > into binomial p and the Poisson lambda The way I did this > > is in the > > > attached Excel file. You will see the binomial p is a > > lousy 0.0003, > > > which converts into a Poisson lambda or Garfield's > constant of 2.15 > > > for the year. This is the probability or the rate articles > > were cited > > > in 1993 on the assumption of probabilistic homogeneity. However, > > > since there is probabilistic heterogeneity, most articles > > have to have > > > a citation rate below Garfield's constant. True to form, > > of the 5000 > > > journals covered that year, 4500 journals were below to > Garfield's > > > colstant. 2.15 is an awful small range to squeeze 4500 > > journals into > > > and expect meaningful quantitative distinctions. > Utilization of a > > > central tendency measure puts one right smack in the > middle of that > > > tight range. Small as this may be, the probabilities and > > lambda were > > > actually much smaller, for Garfield's constant is based on > > the set of > > > articles actually cited that year, i.e., it it truncated on > > the left > > > and does not take into account the articles that could have > > been cited > > > but were not. I do not have the technical or intellectual > > ability to > > > estimate this zero class. I do know that Sir Maurice > > Kendall backed > > > off from the problem when he confronted it in Bradford's > > Law, and who > > > the hell am I compared to Maurice Kendall. I wish that > > somebody would > > > write an article understandable to simpletons on how to make such > > > estimates. From my perspective, this would be one of the most > > > important articles ever written. > > > > > > Sorry for the tirade, but I thought I'd float a few trial > > balloons to > > > be shot down. > > > > > > SB > > > > > > (See attached file: GarConst.xls) > > > > > > > > > > > > > > > > > > > > > > > > > > > Loet Leydesdorff @LISTSERV.UTK.EDU> on > > > 03/04/2006 > > > 07:14:57 AM > > > > > > Please respond to ASIS&T Special Interest Group on Metrics > > > > > > > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > > > > > > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > > > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > > > Rodriguez, and Herbert > > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > > 9 Jan 2006 > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > Dear colleagues, > > > > > > The idea is interesting. However, there a few problems with this > > > paper. > > > First, the authors should not have used Pearson correlation > > > coefficients to compare the rankings, but rank correlations > > > (Spearman's rho or Kendall's tau). Second, it would have been > > > interesting to have a rank correlation with "total cites" > > > given recent discussions (Bensman). Third, the delineation > > of fields > > > in terms of the ISI subject categories is very questionnable. > > > > > > However, the authors are very clear about their results: "We > > > identified ... > > > , but were unable to recognize a meaningful pattern in the > > results." > > > (p. > > > 9). > > > I don't understand why one should then multiply the one > > measure with > > > the other. What does multiplication to the error? > > > > > > Does one of you know a place where the ISI subject categories are > > > justified? > > > How are they produced? People seem to use them > increasingly both in > > > evaluation and research practices, but I have never been able to > > > reproduce them using journal citation measures. > > > > > > With best wishes, > > > > > > > > > Loet > > > > > > > > > ________________________________ > > > Loet Leydesdorff > > > Amsterdam School of Communications Research (ASCoR), > > Kloveniersburgwal > > > 48, 1012 CX Amsterdam. > > > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > > loet at leydesdorff.net ; > > > http://www.leydesdorff.net/ > > > > > > > > > > > > > -----Original Message----- > > > > From: ASIS&T Special Interest Group on Metrics > > > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of > Eugene Garfield > > > > Sent: Friday, March 03, 2006 6:37 PM > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > > > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > > and Herbert > > > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > > > 9 Jan 2006 > > > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > Further to yesterday's posting, "Prestige is factored > > into journal > > > > ratings", here is another interesting and informative article > > > > > > > > FULL TEXT AVAILABLE AT : > > > > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf > > > > > > > > email: {jbollen, marko, herbertv}@lanl.gov > > > > > > > > TITLE : Journal Status > > > > > > > > AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert > > > Van de Sompel > > > > > > > > SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > > > > > Abstract > > > > The status of an actor in a social context is commonly > defined in > > > > terms of two factors: the total number of endorsements > the actor > > > > receives from other actors and the prestige of the > > > endorsing actors. > > > > These two factors indicate the distinction between > popularity and > > > > expert appreciation of the actor, respectively. We refer to > > > the former > > > > as popularity and to the latter as prestige. These notions of > > > > popularity and prestige also apply to the domain of scholarly > > > > assessment. The ISI Impact Factor (ISI IF) is defined > as the mean > > > > number of citations a journal receives over a 2 year > > > period. By merely > > > > counting the amount of citations and disregarding the > > > prestige of the > > > > citing journals, the ISI IF is a metric of popularity, not of > > > > prestige. We demonstrate how a weighted version of the popular > > > > PageRank algorithm can be used to obtain a metric that reflects > > > > prestige. We contrast the rankings of journals according to > > > their ISI > > > > IF and their weighted PageRank, and we provide an analysis that > > > > reveals both significant overlaps and differences. > > > > Furthermore, we introduce the Y-factor which is a simple > > > combination > > > > of both the ISI IF and the weighted PageRank, and find that the > > > > resulting journal rankings correspond well to a general > > > understanding > > > > of journal status. > > > > > > > > > > > > ______________________________________________ > > > > > > > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > FULL TEXT AVAILABLE AT : > > > > > > > > > > http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR > > > > http://guide.labanimal.com/news/2006/060213/full/439770a.html > > > > > > > > > > > > Philip Ball : p.ball at nature.com > > > > www.philipball.com > > > > > > > > Title: Prestige is factored into journal ratings > > > > > > > > Author(s): Ball P > > > > > > > > Source: NATURE 439 (7078): 770-771 FEB 16 2006 > > > > > > > > Document Type: News Item Language: English > > > > Cited References: 0 Times Cited: 0 > > > > > > > > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 > > > CRINAN ST, > > > > LONDON > > > > N1 9XW, ENGLAND > > > > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA > > > > > > > > ISSN: 0028-0836 > > > > > > > > > > > > > From loet at LEYDESDORFF.NET Tue Mar 7 14:09:46 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Tue, 7 Mar 2006 20:09:46 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: Dear colleagues: This is for the SCI 2004 (5967 journals): Total Variance Explained Component Rotation Sums of Squared Loadings Total % of Variance Cumulative % 1 2.450 40.833 40.833 2 2.335 38.921 79.754 Extraction Method: Principal Component Analysis. Thus, the two factors are approximately equal, but orthogonal! With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook.jpg Type: image/jpeg Size: 19534 bytes Desc: not available URL: From loet at LEYDESDORFF.NET Tue Mar 7 16:27:21 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Tue, 7 Mar 2006 22:27:21 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: Dear Stephen, Journals are different in size, prestige, etc., but they are also differently positioned in networks which differ in terms of the densities of the graphs. You are mainly interested in stratification within one graph/discipline, that is, chemistry. For example, you are interested in the differences between review journals and article journals within this set and in the specific position of elite journals of the American professional association. I should add that you use the Library of Congress classification. I am interested also in the differences among the graphs representing specialties and disciplines. Impact factors are used as measures in a hierarchy, but they should be corrected for these network properties. Our colleagues in Leiden tried this long ago, but failed to get their measure accepted. In the meantime, we have Freeman's four centrality measures. I promised a paper in Leuven at the indicators meeting using these measures on the journal set. I guess that the impact factor and the degree measure will correlate. (I am particularly interested in the betweenness measure as an indicator of interdisciplinarity.) The problem with the differentiation is that one then needs subject categories. You have explained to me that the subsets are fuzzy. This is where this discussion started. Bollen et al. use the ISI categories and have fuzzy results; perhaps, not only for this reason. Nobody on this list seems to be able to explain how ISI decides to organize journals into categories. Perhaps, this is better so because then everybody can use them in research and research evaluation. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman > Sent: Tuesday, March 07, 2006 7:54 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Loet, > Let me offer a suggestion on why impact factor is so > separated from the other variables on your graphic. The > journals causing most of the variance in impact factor are > review journals. Their high average citation results from > their function. The other variable, in which review journals > are a significant source of variance, is faculty score, which > has the two key review journals among its top 12 or 10%, > thereby indicating the importance of review journals to > scientists. I do notice that faculty score is the closest of > the other variables to impact factor on your graphic. > Outlier analysis has revealed that review journals have > higher library use than warranted by their size--a further > indication of their functional importance. > > My conclusions are thus. Garfield was correct in his > employment of impact factor to identify the important review > journals among the small journals whose significance would be > otherwise inundated by size. The separation of impact factor > from the other variables on your graphic is due mainly due to > differing function of the journals causing the variance in > impact factor. > It is not differing prestige, because the key review journals > are mostly US association journals like the key research journals. > > There is a role played by fuzzy sets. The higher citation > rates of biochemistry journals forces these journals to the > top in impact factor, giving a further reason for the > isolation of this variable on your graphic. > Most of the biochemistry citations are exogenous to the > interests of the LSU chemistry faculty and patrons of the UI > Chemistry Library. This is seen from the fact that > biochemistry journals rank low on faculty score and library > use but high on both the citation measures. > > How do these explanations play with you? > > SB > > > > > > > Loet Leydesdorff @listserv.utk.edu> on > 03/07/2006 > 11:38:56 AM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at listserv.utk.edu > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > Rodriguez, and Herbert > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Dear Stephen, > > It seems that we agree that there is a strong factor > correlating size, number of citations, number of > publications, faculty appreciation, etc., which is orthogonal > to the impact factor (when using your dataset). Within your > dataset, you can make these statements about the quality of > the impact factor because it is relatively a disciplinarily > homogenous set of chemistry journals. This would break down > in the case of more mixed sets. > > I can do it for the complete set, but I don't have faculty > appreciations; only total citations, nr of publications and > impact factors. The expectation is that I find the two > factors again because impact factors are not really different > from c/p ratios (with a delay), and c/p ratios are the > quotients of two of the indicators on the other axis. c/p > ratios can be expected to correlate highly with IFs, but not > with c or p themselves. > > With best wishes, > > > Loet > > ________________________________ > Loet Leydesdorff > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal 48, 1012 CX Amsterdam. > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; http://www.leydesdorff.net/ > > > > > -----Original Message----- > > From: ASIS&T Special Interest Group on Metrics > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman > > Sent: Tuesday, March 07, 2006 5:40 PM > > To: SIGMETRICS at LISTSERV.UTK.EDU > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and > > Herbert Van de Sompel "Journal Status" > > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > Loet, > > I looked at your picture and will try to interpret it in my > terms. I > > see that LSU faculty score, UIUC use, and total citations are > > clustered tightly together, whereas impact factor is isolated and > > apart. Scientist ratings, library use, and total citations are > > heavily influenced by size. For example, when scientist ratings of > > the quality of chemistry programs are correlated with the average > > citations per faculty, the correlation is 0.81, but when these same > > ratings are correlated with total program citations, the > correlation > > rises to 0.91. In his writings Garfield often commented upon the > > relationship of size to signifcance, calling upon national > journals to > > consolidate into regional journals to gain added heft. I > notice that > > lately there has been a move among European associations to > > consolidate their national journals into European ones. > > > > Library use and total citations are size dependent in two > > senses--physical size of journals and length of backfile. > > > > Impact factor deliberately controls for both components of size. > > Moreover, problems with classifying sources into citable and > > non-citable causes it to have more random error, and this > random error > > operates within a tightly constricted range, causing > greater scatter > > in plots. However, here, also, the distributions are so > exponentially > > skewed, that it readily identifies those journals, whose > articles have > > a higher probability of being cited for reasons of > function, prestige, > > or both. The correlations of total citations with impact factor > > particularly after the removal of review journal outliers > are so high > > that a cruder method measuring association within broad categories > > that is more robust against error may reveal a high degree > of overlap > > at the top for all four measures. I have to test for this. > > > > Does this clarify matters for you? > > > > SB > > > > > > > > > > > > > > Loet Leydesdorff @LISTSERV.UTK.EDU> on > > 03/07/2006 02:00:30 AM > > > > Please respond to ASIS&T Special Interest Group on Metrics > > > > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > > Rodriguez, and Herbert > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > Thanks, Stephen: I find this very clear. > > > > But I thought that you had not found such a high > correlation between > > IFs and total cites. When I compute this correlation over the SCI > > 2004, I find r = > > .405 and Spearman's rho = .726. Not too bad! (For the SoSCI > r = 0.642 > > and rho = .746. Of course, everything is highly significant > because of > > the high > > numbers.) > > > > Nevertheless, the picture which I sent you yesterday clearly shows > > that there are two factors involved: total cites are like > total number > > of publications and faculty assessment based on properties of the > > journal, that is, the nodes of the networks, while IF's and > c/p ratios > > are properties (e.g., > > averages) of links. > > > > Network parameters are different from node attributes. Both are > > interesting, but different. > > > > With kind regards, > > > > > > Loet > > > > ________________________________ > > Loet Leydesdorff > > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal > > 48, 1012 CX Amsterdam. > > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; > > http://www.leydesdorff.net/ > > > > > > > > > -----Original Message----- > > > From: ASIS&T Special Interest Group on Metrics > > > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Stephen > J Bensman > > > Sent: Monday, March 06, 2006 11:35 PM > > > To: SIGMETRICS at listserv.utk.edu > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and > > > Herbert Van de Sompel "Journal Status" > > > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > > > > > > > > > > Loet, > > > You and I are perhaps looking at two different aspects of > > the problem. > > > I am looking at it from the perspective of a librarian trying to > > > decide which journals should be provided with permanent > access on a > > > subscription basis and which should be accessed through > > some form of > > > intermittent document delivery. Therefore, I am interested > > not only > > > in prestige but also in functionality, i.e., what > function does the > > > journal serve--reporting of research, reviewing of literature, > > > assistance in teaching, provision of current news, etc. Citation > > > measures either capture one facet of functionality--total > citations > > > seem to capture reporting of research, impact factor, review > > > literature--or fail to capture the functionality at all, i.e., > > > teaching or reporting of current news. Total citations > > cannot capture > > > the review literature, because review journals are usually > > very small > > > even though highly rated by scientists, but since impact factor > > > captures both review literature and current research > > > significance--which is usually the same as historical historical > > > research significance due the stability of patterns--and the > > > correlation of total citations with impact factor is high > enough so > > > that journals high on both can be captured in a broad > > category robust > > > against random error, it seems to me that impact factor > can capture > > > two facets of functionality unlike total citations, which > can only > > > capture one. The hypothesis remains to be tested. > > > > > > Prestige appears to operate separately from functionality. > > > The greatest cause of variance in all four measures is > > their belongine > > > to the category of US association journals. The journals mf the > > > American Chemical Society are dominant on all four > > measures. Through > > > various evaluations of US research-doctorate programs by > > peer ratings > > > and citations, I can trace this dominance to scientists > employed by > > > the traditionally elite US research insitutions. Thus, > > variance and > > > prestige in all four measures is a function of the social > > > stratification system of US scientific institutions. > > > There pemains the question of how do foreign scientists > > relate to the > > > US social stratification system. If they form a part of it, the > > > foreigners can use ISI citations for evaluation and other > > puposes. If > > > they do not form part of it, then foreigners using ISI > > citations may > > > only be rating themselves by how much their work is being > > accepted by > > > the scientists within this system. I have no answer to > > this question. > > > > > > I suppose that now you are more confused than ever. I > did my best, > > > but it is complicated as all hell. > > > > > > SB > > > > > > > > > > > > > > > Loet Leydesdorff @listserv.utk.edu> on > > > 03/06/2006 > > > 03:12:54 PM > > > > > > Please respond to ASIS&T Special Interest Group on Metrics > > > > > > > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > > > > > > > > > To: SIGMETRICS at listserv.utk.edu > > > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > > > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > > > Rodriguez, and Herbert > > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > > 9 Jan 2006 > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > Dear Stephen, > > > > > > I apologize if I dragged you into the discussion with improper > > > argument, but I did not want to mention your idea of using > > total cites > > > as an indicator without providing a proper reference. > > > > > > The reasoning in your posting is difficult for me to > follow, but I > > > look forward to reading the full paper. My experience is > > that reading > > > the full paper, one begins to understand. I found your previous > > > argument about using total cites very convincing because of > > its high > > > correlation with faculty ratings and its orthogonality to > > the impact > > > factor. It seemed to me that the impact factor measures > > something very > > > different from the prestige of a journal. > > > (Embedded image moved to file: pic17086.gif) Figure 1: > > > Component plot in rotated space (sources: JCR, 1993; > Bensman, 2001; > > > forthcoming; Bensman & Wilder, 1998). > > > > > > > > > From: Visualization of the Citation Impact Environments of > > Scientific > > > Journals: An online mapping exercise, Journal of the > > American Society > > > for Information Science and Technology (forthcoming). > > > . > > > > > > > > > With best wishes, > > > > > > > > > Loet > > > > > > ________________________________ > > > Loet Leydesdorff > > > Amsterdam School of Communications Research (ASCoR), > > Kloveniersburgwal > > > 48, 1012 CX Amsterdam. > > > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > > loet at leydesdorff.net ; > > > http://www.leydesdorff.net/ > > > > > > > > > > > > > -----Original Message----- > > > > From: ASIS&T Special Interest Group on Metrics > > > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen > > J Bensman > > > > Sent: Monday, March 06, 2006 6:41 PM > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and > > > > Herbert Van de Sompel "Journal Status" > > > > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > > > > > > > > > > > > > > > > Loet, > > > > I see that you have once again taken my name in vain and > > > again given > > > > me the opportunity to spout my ideas on SIGMETRICS. I must > > > admit that > > > > I have not read the paper you discuss, because my doctor > > warned me > > > > against reading too many such papers, since I am fairly > close to > > > > OD-ing on them. > > > > However, the conclusions you mention do seem a little peculiar. > > > > > > > > Due to detailed study of Gene Garfield's development and > > > utilization > > > > of impact factor, I am coming to change my mind on this measure > > > > somewhat. It is for rather complicated reasons, which I > > > shall try to > > > > explain below. > > > > > > > > In general I think that there is too much random error in > > citation > > > > data for the utilization of such precise techniques as > > > > correlation--Pearson, Spearman, whatever. Much results > > > from exogenous > > > > citations due to an inability to define precise sets--a logical > > > > consequence of Bradford's Law of Scattering and > Garfield's Law of > > > > Concentration. Impact factor suffers from a further source > > > of error > > > > due to an inability to classify precisely sources into > > citable and > > > > non-citable--something which honest persons can disagree on. > > > > This inability severely affects the denominator of the > > > impact factor > > > > equation. What is therefore needed is a technique that is > > > crude and > > > > robust against such error. I have personally found it in the > > > > chi-square test of independence, which allows the conversion of > > > > citation measures into ordinal variables defined by broad > > > categories. > > > > It also allows one to define the amount of error one is > > willing to > > > > accept, i.e., upper 10% vs. > > > > upper 25%. > > > > > > > > Use of this chi-square test may vindicate impact factor by > > > > demonstrating that it has the same strong relationship > to expert > > > > ratings as do total citations. As a matter of fact, it > may be a > > > > superior measure in that it will not only capture the > > importance of > > > > reseach journals but also of review journals. Close > > > inspection of the > > > > top 10% of the journals recommended by the LSU > chemistry faculty > > > > reveals it to be a balanced mix of research journals, review, > > > > journals, and the main teaching journal of chemistry. > > > > In other words, most facets of journal importance are > > > captured by this > > > > measure, whereas total citations captures mainly research, > > > and impact > > > > factor captures chiefly the review journals. However, > > > broadening the > > > > categories may cause impact factor to capture both research > > > and review > > > > though not the teaching facet. In any case I am going to > > > test this in > > > > the revision of the JASIST paper I am now engaged in. > > > > > > > > Impact factor has the ability to do this for the very > > > reasons Seglen > > > > denounces it. His main case against is based on the > > > reasoning of the > > > > law of error and the role of the arithmetic mean in this > > law. This > > > > requires the normal distribution for the arithmetic > mean to be an > > > > accurate estimate of central tendency. However, due to > > the highly > > > > skewed distributions with which we deal, the arithmetic > > > mean is always > > > > way above the other estimates of central tendency such as > > > the median > > > > or the geometric mean due to the high degree of variance > > caused the > > > > dominant observations. Seglen's reasoning collapses once > > > one realizes > > > > that a journal's or scientist's importance is not measured > > > by central > > > > tendency but by the variance caused by the few > important articles > > > > published by the journal or scientist. > > > > Therefore, scientific importance is the result of > > variance and not > > > > central tendency. The arithmetic mean, which impact factor > > > attempts > > > > to estimate, better captures the variance. > > > > > > > > To demonstrate, I have converted Garfield's constant for > > > the year 1993 > > > > into binomial p and the Poisson lambda The way I did this > > > is in the > > > > attached Excel file. You will see the binomial p is a > > > lousy 0.0003, > > > > which converts into a Poisson lambda or Garfield's > > constant of 2.15 > > > > for the year. This is the probability or the rate articles > > > were cited > > > > in 1993 on the assumption of probabilistic homogeneity. > However, > > > > since there is probabilistic heterogeneity, most articles > > > have to have > > > > a citation rate below Garfield's constant. True to form, > > > of the 5000 > > > > journals covered that year, 4500 journals were below to > > Garfield's > > > > colstant. 2.15 is an awful small range to squeeze 4500 > > > journals into > > > > and expect meaningful quantitative distinctions. > > Utilization of a > > > > central tendency measure puts one right smack in the > > middle of that > > > > tight range. Small as this may be, the probabilities and > > > lambda were > > > > actually much smaller, for Garfield's constant is based on > > > the set of > > > > articles actually cited that year, i.e., it it truncated on > > > the left > > > > and does not take into account the articles that could have > > > been cited > > > > but were not. I do not have the technical or intellectual > > > ability to > > > > estimate this zero class. I do know that Sir Maurice > > > Kendall backed > > > > off from the problem when he confronted it in Bradford's > > > Law, and who > > > > the hell am I compared to Maurice Kendall. I wish that > > > somebody would > > > > write an article understandable to simpletons on how to > make such > > > > estimates. From my perspective, this would be one of the most > > > > important articles ever written. > > > > > > > > Sorry for the tirade, but I thought I'd float a few trial > > > balloons to > > > > be shot down. > > > > > > > > SB > > > > > > > > (See attached file: GarConst.xls) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Loet Leydesdorff @LISTSERV.UTK.EDU> on > > > > 03/04/2006 > > > > 07:14:57 AM > > > > > > > > Please respond to ASIS&T Special Interest Group on Metrics > > > > > > > > > > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > > > > > > > > > > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > > > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > > > > > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > > > > Rodriguez, and Herbert > > > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > > > 9 Jan 2006 > > > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > Dear colleagues, > > > > > > > > The idea is interesting. However, there a few problems > with this > > > > paper. > > > > First, the authors should not have used Pearson correlation > > > > coefficients to compare the rankings, but rank correlations > > > > (Spearman's rho or Kendall's tau). Second, it would have been > > > > interesting to have a rank correlation with "total cites" > > > > given recent discussions (Bensman). Third, the delineation > > > of fields > > > > in terms of the ISI subject categories is very questionnable. > > > > > > > > However, the authors are very clear about their results: "We > > > > identified ... > > > > , but were unable to recognize a meaningful pattern in the > > > results." > > > > (p. > > > > 9). > > > > I don't understand why one should then multiply the one > > > measure with > > > > the other. What does multiplication to the error? > > > > > > > > Does one of you know a place where the ISI subject > categories are > > > > justified? > > > > How are they produced? People seem to use them > > increasingly both in > > > > evaluation and research practices, but I have never > been able to > > > > reproduce them using journal citation measures. > > > > > > > > With best wishes, > > > > > > > > > > > > Loet > > > > > > > > > > > > ________________________________ > > > > Loet Leydesdorff > > > > Amsterdam School of Communications Research (ASCoR), > > > Kloveniersburgwal > > > > 48, 1012 CX Amsterdam. > > > > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > > > loet at leydesdorff.net ; > > > > http://www.leydesdorff.net/ > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: ASIS&T Special Interest Group on Metrics > > > > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of > > Eugene Garfield > > > > > Sent: Friday, March 03, 2006 6:37 PM > > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > > > > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > > > and Herbert > > > > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > > > > 9 Jan 2006 > > > > > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > > Further to yesterday's posting, "Prestige is factored > > > into journal > > > > > ratings", here is another interesting and informative article > > > > > > > > > > FULL TEXT AVAILABLE AT : > > > > > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf > > > > > > > > > > email: {jbollen, marko, herbertv}@lanl.gov > > > > > > > > > > TITLE : Journal Status > > > > > > > > > > AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert > > > > Van de Sompel > > > > > > > > > > SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > > > > > > > Abstract > > > > > The status of an actor in a social context is commonly > > defined in > > > > > terms of two factors: the total number of endorsements > > the actor > > > > > receives from other actors and the prestige of the > > > > endorsing actors. > > > > > These two factors indicate the distinction between > > popularity and > > > > > expert appreciation of the actor, respectively. We refer to > > > > the former > > > > > as popularity and to the latter as prestige. These notions of > > > > > popularity and prestige also apply to the domain of scholarly > > > > > assessment. The ISI Impact Factor (ISI IF) is defined > > as the mean > > > > > number of citations a journal receives over a 2 year > > > > period. By merely > > > > > counting the amount of citations and disregarding the > > > > prestige of the > > > > > citing journals, the ISI IF is a metric of popularity, not of > > > > > prestige. We demonstrate how a weighted version of > the popular > > > > > PageRank algorithm can be used to obtain a metric > that reflects > > > > > prestige. We contrast the rankings of journals according to > > > > their ISI > > > > > IF and their weighted PageRank, and we provide an > analysis that > > > > > reveals both significant overlaps and differences. > > > > > Furthermore, we introduce the Y-factor which is a simple > > > > combination > > > > > of both the ISI IF and the weighted PageRank, and > find that the > > > > > resulting journal rankings correspond well to a general > > > > understanding > > > > > of journal status. > > > > > > > > > > > > > > > ______________________________________________ > > > > > > > > > > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > > FULL TEXT AVAILABLE AT : > > > > > > > > > > > > > > > http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR > > > > > http://guide.labanimal.com/news/2006/060213/full/439770a.html > > > > > > > > > > > > > > > Philip Ball : p.ball at nature.com > > > > > www.philipball.com > > > > > > > > > > Title: Prestige is factored into journal ratings > > > > > > > > > > Author(s): Ball P > > > > > > > > > > Source: NATURE 439 (7078): 770-771 FEB 16 2006 > > > > > > > > > > Document Type: News Item Language: English > > > > > Cited References: 0 Times Cited: 0 > > > > > > > > > > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 > > > > CRINAN ST, > > > > > LONDON > > > > > N1 9XW, ENGLAND > > > > > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA > > > > > > > > > > ISSN: 0028-0836 > > > > > > > > > > > > > > > > > > > From notsjb at LSU.EDU Tue Mar 7 17:33:07 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Tue, 7 Mar 2006 16:33:07 -0600 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: Loet, First, before I do anything, I first try to define rational subject sets as much as possible. Otherwise you get so many outliers and interaction effects that you obtain nonsensical results. Therefore I work only within defined disciplines. Second, I work with LC classification, because libraries are orgnanized that way, and I have ready made sets. The LC system is not the best of systems, but it is perhaps the most detailed and comprehensive. Moreover, it is understood and used by most people in my field. However, there are reams of articles complaining about its illogic, etc. Third, I really do not think that anybody can come up with a single classification system that will satisfy everybody. Libraries found it impossible to establish collection development policies, because academic departments cannot define their own subject parameters. Requests for them to do so always wound up in ugly cat fights, in which librarians were used as scatching posts by the faculty. I live in an imperfect world and just accept it. There are about as many ways to look at subjects as there are people in the world. Fourth, I do not think that subjects can be defined mathematically. Perhaps I am old fashioned, but the best way to do it is the old fashioned way. Develop some sort of philosophical and logical system, then look at the item, and see how it fits within your subjective logical system. Keynes called it analogy and comparison. Using citations to classify materials works only for the small set covered by ISI, and this only a small sample of human knowledge Perhaps we should continue this offline and not bother a lot of people with this. SB PS I use ISI subject categories and have found them very good. But then I am used to using bad classification systems like LC and the DDC. Loet Leydesdorff @LISTSERV.UTK.EDU> on 03/07/2006 03:27:21 PM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Dear Stephen, Journals are different in size, prestige, etc., but they are also differently positioned in networks which differ in terms of the densities of the graphs. You are mainly interested in stratification within one graph/discipline, that is, chemistry. For example, you are interested in the differences between review journals and article journals within this set and in the specific position of elite journals of the American professional association. I should add that you use the Library of Congress classification. I am interested also in the differences among the graphs representing specialties and disciplines. Impact factors are used as measures in a hierarchy, but they should be corrected for these network properties. Our colleagues in Leiden tried this long ago, but failed to get their measure accepted. In the meantime, we have Freeman's four centrality measures. I promised a paper in Leuven at the indicators meeting using these measures on the journal set. I guess that the impact factor and the degree measure will correlate. (I am particularly interested in the betweenness measure as an indicator of interdisciplinarity.) The problem with the differentiation is that one then needs subject categories. You have explained to me that the subsets are fuzzy. This is where this discussion started. Bollen et al. use the ISI categories and have fuzzy results; perhaps, not only for this reason. Nobody on this list seems to be able to explain how ISI decides to organize journals into categories. Perhaps, this is better so because then everybody can use them in research and research evaluation. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman > Sent: Tuesday, March 07, 2006 7:54 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Loet, > Let me offer a suggestion on why impact factor is so > separated from the other variables on your graphic. The > journals causing most of the variance in impact factor are > review journals. Their high average citation results from > their function. The other variable, in which review journals > are a significant source of variance, is faculty score, which > has the two key review journals among its top 12 or 10%, > thereby indicating the importance of review journals to > scientists. I do notice that faculty score is the closest of > the other variables to impact factor on your graphic. > Outlier analysis has revealed that review journals have > higher library use than warranted by their size--a further > indication of their functional importance. > > My conclusions are thus. Garfield was correct in his > employment of impact factor to identify the important review > journals among the small journals whose significance would be > otherwise inundated by size. The separation of impact factor > from the other variables on your graphic is due mainly due to > differing function of the journals causing the variance in > impact factor. > It is not differing prestige, because the key review journals > are mostly US association journals like the key research journals. > > There is a role played by fuzzy sets. The higher citation > rates of biochemistry journals forces these journals to the > top in impact factor, giving a further reason for the > isolation of this variable on your graphic. > Most of the biochemistry citations are exogenous to the > interests of the LSU chemistry faculty and patrons of the UI > Chemistry Library. This is seen from the fact that > biochemistry journals rank low on faculty score and library > use but high on both the citation measures. > > How do these explanations play with you? > > SB > > > > > > > Loet Leydesdorff @listserv.utk.edu> on > 03/07/2006 > 11:38:56 AM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at listserv.utk.edu > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > Rodriguez, and Herbert > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Dear Stephen, > > It seems that we agree that there is a strong factor > correlating size, number of citations, number of > publications, faculty appreciation, etc., which is orthogonal > to the impact factor (when using your dataset). Within your > dataset, you can make these statements about the quality of > the impact factor because it is relatively a disciplinarily > homogenous set of chemistry journals. This would break down > in the case of more mixed sets. > > I can do it for the complete set, but I don't have faculty > appreciations; only total citations, nr of publications and > impact factors. The expectation is that I find the two > factors again because impact factors are not really different > from c/p ratios (with a delay), and c/p ratios are the > quotients of two of the indicators on the other axis. c/p > ratios can be expected to correlate highly with IFs, but not > with c or p themselves. > > With best wishes, > > > Loet > > ________________________________ > Loet Leydesdorff > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal 48, 1012 CX Amsterdam. > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; http://www.leydesdorff.net/ > > > > > -----Original Message----- > > From: ASIS&T Special Interest Group on Metrics > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman > > Sent: Tuesday, March 07, 2006 5:40 PM > > To: SIGMETRICS at LISTSERV.UTK.EDU > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and > > Herbert Van de Sompel "Journal Status" > > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > Loet, > > I looked at your picture and will try to interpret it in my > terms. I > > see that LSU faculty score, UIUC use, and total citations are > > clustered tightly together, whereas impact factor is isolated and > > apart. Scientist ratings, library use, and total citations are > > heavily influenced by size. For example, when scientist ratings of > > the quality of chemistry programs are correlated with the average > > citations per faculty, the correlation is 0.81, but when these same > > ratings are correlated with total program citations, the > correlation > > rises to 0.91. In his writings Garfield often commented upon the > > relationship of size to signifcance, calling upon national > journals to > > consolidate into regional journals to gain added heft. I > notice that > > lately there has been a move among European associations to > > consolidate their national journals into European ones. > > > > Library use and total citations are size dependent in two > > senses--physical size of journals and length of backfile. > > > > Impact factor deliberately controls for both components of size. > > Moreover, problems with classifying sources into citable and > > non-citable causes it to have more random error, and this > random error > > operates within a tightly constricted range, causing > greater scatter > > in plots. However, here, also, the distributions are so > exponentially > > skewed, that it readily identifies those journals, whose > articles have > > a higher probability of being cited for reasons of > function, prestige, > > or both. The correlations of total citations with impact factor > > particularly after the removal of review journal outliers > are so high > > that a cruder method measuring association within broad categories > > that is more robust against error may reveal a high degree > of overlap > > at the top for all four measures. I have to test for this. > > > > Does this clarify matters for you? > > > > SB > > > > > > > > > > > > > > Loet Leydesdorff @LISTSERV.UTK.EDU> on > > 03/07/2006 02:00:30 AM > > > > Please respond to ASIS&T Special Interest Group on Metrics > > > > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > > Rodriguez, and Herbert > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > 9 Jan 2006 > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > Thanks, Stephen: I find this very clear. > > > > But I thought that you had not found such a high > correlation between > > IFs and total cites. When I compute this correlation over the SCI > > 2004, I find r = > > .405 and Spearman's rho = .726. Not too bad! (For the SoSCI > r = 0.642 > > and rho = .746. Of course, everything is highly significant > because of > > the high > > numbers.) > > > > Nevertheless, the picture which I sent you yesterday clearly shows > > that there are two factors involved: total cites are like > total number > > of publications and faculty assessment based on properties of the > > journal, that is, the nodes of the networks, while IF's and > c/p ratios > > are properties (e.g., > > averages) of links. > > > > Network parameters are different from node attributes. Both are > > interesting, but different. > > > > With kind regards, > > > > > > Loet > > > > ________________________________ > > Loet Leydesdorff > > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal > > 48, 1012 CX Amsterdam. > > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; > > http://www.leydesdorff.net/ > > > > > > > > > -----Original Message----- > > > From: ASIS&T Special Interest Group on Metrics > > > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Stephen > J Bensman > > > Sent: Monday, March 06, 2006 11:35 PM > > > To: SIGMETRICS at listserv.utk.edu > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and > > > Herbert Van de Sompel "Journal Status" > > > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > > > > > > > > > > Loet, > > > You and I are perhaps looking at two different aspects of > > the problem. > > > I am looking at it from the perspective of a librarian trying to > > > decide which journals should be provided with permanent > access on a > > > subscription basis and which should be accessed through > > some form of > > > intermittent document delivery. Therefore, I am interested > > not only > > > in prestige but also in functionality, i.e., what > function does the > > > journal serve--reporting of research, reviewing of literature, > > > assistance in teaching, provision of current news, etc. Citation > > > measures either capture one facet of functionality--total > citations > > > seem to capture reporting of research, impact factor, review > > > literature--or fail to capture the functionality at all, i.e., > > > teaching or reporting of current news. Total citations > > cannot capture > > > the review literature, because review journals are usually > > very small > > > even though highly rated by scientists, but since impact factor > > > captures both review literature and current research > > > significance--which is usually the same as historical historical > > > research significance due the stability of patterns--and the > > > correlation of total citations with impact factor is high > enough so > > > that journals high on both can be captured in a broad > > category robust > > > against random error, it seems to me that impact factor > can capture > > > two facets of functionality unlike total citations, which > can only > > > capture one. The hypothesis remains to be tested. > > > > > > Prestige appears to operate separately from functionality. > > > The greatest cause of variance in all four measures is > > their belongine > > > to the category of US association journals. The journals mf the > > > American Chemical Society are dominant on all four > > measures. Through > > > various evaluations of US research-doctorate programs by > > peer ratings > > > and citations, I can trace this dominance to scientists > employed by > > > the traditionally elite US research insitutions. Thus, > > variance and > > > prestige in all four measures is a function of the social > > > stratification system of US scientific institutions. > > > There pemains the question of how do foreign scientists > > relate to the > > > US social stratification system. If they form a part of it, the > > > foreigners can use ISI citations for evaluation and other > > puposes. If > > > they do not form part of it, then foreigners using ISI > > citations may > > > only be rating themselves by how much their work is being > > accepted by > > > the scientists within this system. I have no answer to > > this question. > > > > > > I suppose that now you are more confused than ever. I > did my best, > > > but it is complicated as all hell. > > > > > > SB > > > > > > > > > > > > > > > Loet Leydesdorff @listserv.utk.edu> on > > > 03/06/2006 > > > 03:12:54 PM > > > > > > Please respond to ASIS&T Special Interest Group on Metrics > > > > > > > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > > > > > > > > > To: SIGMETRICS at listserv.utk.edu > > > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > > > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > > > Rodriguez, and Herbert > > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > > 9 Jan 2006 > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > Dear Stephen, > > > > > > I apologize if I dragged you into the discussion with improper > > > argument, but I did not want to mention your idea of using > > total cites > > > as an indicator without providing a proper reference. > > > > > > The reasoning in your posting is difficult for me to > follow, but I > > > look forward to reading the full paper. My experience is > > that reading > > > the full paper, one begins to understand. I found your previous > > > argument about using total cites very convincing because of > > its high > > > correlation with faculty ratings and its orthogonality to > > the impact > > > factor. It seemed to me that the impact factor measures > > something very > > > different from the prestige of a journal. > > > (Embedded image moved to file: pic17086.gif) Figure 1: > > > Component plot in rotated space (sources: JCR, 1993; > Bensman, 2001; > > > forthcoming; Bensman & Wilder, 1998). > > > > > > > > > From: Visualization of the Citation Impact Environments of > > Scientific > > > Journals: An online mapping exercise, Journal of the > > American Society > > > for Information Science and Technology (forthcoming). > > > . > > > > > > > > > With best wishes, > > > > > > > > > Loet > > > > > > ________________________________ > > > Loet Leydesdorff > > > Amsterdam School of Communications Research (ASCoR), > > Kloveniersburgwal > > > 48, 1012 CX Amsterdam. > > > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > > loet at leydesdorff.net ; > > > http://www.leydesdorff.net/ > > > > > > > > > > > > > -----Original Message----- > > > > From: ASIS&T Special Interest Group on Metrics > > > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen > > J Bensman > > > > Sent: Monday, March 06, 2006 6:41 PM > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and > > > > Herbert Van de Sompel "Journal Status" > > > > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > > > > > > > > > > > > > > > > Loet, > > > > I see that you have once again taken my name in vain and > > > again given > > > > me the opportunity to spout my ideas on SIGMETRICS. I must > > > admit that > > > > I have not read the paper you discuss, because my doctor > > warned me > > > > against reading too many such papers, since I am fairly > close to > > > > OD-ing on them. > > > > However, the conclusions you mention do seem a little peculiar. > > > > > > > > Due to detailed study of Gene Garfield's development and > > > utilization > > > > of impact factor, I am coming to change my mind on this measure > > > > somewhat. It is for rather complicated reasons, which I > > > shall try to > > > > explain below. > > > > > > > > In general I think that there is too much random error in > > citation > > > > data for the utilization of such precise techniques as > > > > correlation--Pearson, Spearman, whatever. Much results > > > from exogenous > > > > citations due to an inability to define precise sets--a logical > > > > consequence of Bradford's Law of Scattering and > Garfield's Law of > > > > Concentration. Impact factor suffers from a further source > > > of error > > > > due to an inability to classify precisely sources into > > citable and > > > > non-citable--something which honest persons can disagree on. > > > > This inability severely affects the denominator of the > > > impact factor > > > > equation. What is therefore needed is a technique that is > > > crude and > > > > robust against such error. I have personally found it in the > > > > chi-square test of independence, which allows the conversion of > > > > citation measures into ordinal variables defined by broad > > > categories. > > > > It also allows one to define the amount of error one is > > willing to > > > > accept, i.e., upper 10% vs. > > > > upper 25%. > > > > > > > > Use of this chi-square test may vindicate impact factor by > > > > demonstrating that it has the same strong relationship > to expert > > > > ratings as do total citations. As a matter of fact, it > may be a > > > > superior measure in that it will not only capture the > > importance of > > > > reseach journals but also of review journals. Close > > > inspection of the > > > > top 10% of the journals recommended by the LSU > chemistry faculty > > > > reveals it to be a balanced mix of research journals, review, > > > > journals, and the main teaching journal of chemistry. > > > > In other words, most facets of journal importance are > > > captured by this > > > > measure, whereas total citations captures mainly research, > > > and impact > > > > factor captures chiefly the review journals. However, > > > broadening the > > > > categories may cause impact factor to capture both research > > > and review > > > > though not the teaching facet. In any case I am going to > > > test this in > > > > the revision of the JASIST paper I am now engaged in. > > > > > > > > Impact factor has the ability to do this for the very > > > reasons Seglen > > > > denounces it. His main case against is based on the > > > reasoning of the > > > > law of error and the role of the arithmetic mean in this > > law. This > > > > requires the normal distribution for the arithmetic > mean to be an > > > > accurate estimate of central tendency. However, due to > > the highly > > > > skewed distributions with which we deal, the arithmetic > > > mean is always > > > > way above the other estimates of central tendency such as > > > the median > > > > or the geometric mean due to the high degree of variance > > caused the > > > > dominant observations. Seglen's reasoning collapses once > > > one realizes > > > > that a journal's or scientist's importance is not measured > > > by central > > > > tendency but by the variance caused by the few > important articles > > > > published by the journal or scientist. > > > > Therefore, scientific importance is the result of > > variance and not > > > > central tendency. The arithmetic mean, which impact factor > > > attempts > > > > to estimate, better captures the variance. > > > > > > > > To demonstrate, I have converted Garfield's constant for > > > the year 1993 > > > > into binomial p and the Poisson lambda The way I did this > > > is in the > > > > attached Excel file. You will see the binomial p is a > > > lousy 0.0003, > > > > which converts into a Poisson lambda or Garfield's > > constant of 2.15 > > > > for the year. This is the probability or the rate articles > > > were cited > > > > in 1993 on the assumption of probabilistic homogeneity. > However, > > > > since there is probabilistic heterogeneity, most articles > > > have to have > > > > a citation rate below Garfield's constant. True to form, > > > of the 5000 > > > > journals covered that year, 4500 journals were below to > > Garfield's > > > > colstant. 2.15 is an awful small range to squeeze 4500 > > > journals into > > > > and expect meaningful quantitative distinctions. > > Utilization of a > > > > central tendency measure puts one right smack in the > > middle of that > > > > tight range. Small as this may be, the probabilities and > > > lambda were > > > > actually much smaller, for Garfield's constant is based on > > > the set of > > > > articles actually cited that year, i.e., it it truncated on > > > the left > > > > and does not take into account the articles that could have > > > been cited > > > > but were not. I do not have the technical or intellectual > > > ability to > > > > estimate this zero class. I do know that Sir Maurice > > > Kendall backed > > > > off from the problem when he confronted it in Bradford's > > > Law, and who > > > > the hell am I compared to Maurice Kendall. I wish that > > > somebody would > > > > write an article understandable to simpletons on how to > make such > > > > estimates. From my perspective, this would be one of the most > > > > important articles ever written. > > > > > > > > Sorry for the tirade, but I thought I'd float a few trial > > > balloons to > > > > be shot down. > > > > > > > > SB > > > > > > > > (See attached file: GarConst.xls) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Loet Leydesdorff @LISTSERV.UTK.EDU> on > > > > 03/04/2006 > > > > 07:14:57 AM > > > > > > > > Please respond to ASIS&T Special Interest Group on Metrics > > > > > > > > > > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > > > > > > > > > > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > > > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > > > > > > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > > > > Rodriguez, and Herbert > > > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > > > 9 Jan 2006 > > > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > Dear colleagues, > > > > > > > > The idea is interesting. However, there a few problems > with this > > > > paper. > > > > First, the authors should not have used Pearson correlation > > > > coefficients to compare the rankings, but rank correlations > > > > (Spearman's rho or Kendall's tau). Second, it would have been > > > > interesting to have a rank correlation with "total cites" > > > > given recent discussions (Bensman). Third, the delineation > > > of fields > > > > in terms of the ISI subject categories is very questionnable. > > > > > > > > However, the authors are very clear about their results: "We > > > > identified ... > > > > , but were unable to recognize a meaningful pattern in the > > > results." > > > > (p. > > > > 9). > > > > I don't understand why one should then multiply the one > > > measure with > > > > the other. What does multiplication to the error? > > > > > > > > Does one of you know a place where the ISI subject > categories are > > > > justified? > > > > How are they produced? People seem to use them > > increasingly both in > > > > evaluation and research practices, but I have never > been able to > > > > reproduce them using journal citation measures. > > > > > > > > With best wishes, > > > > > > > > > > > > Loet > > > > > > > > > > > > ________________________________ > > > > Loet Leydesdorff > > > > Amsterdam School of Communications Research (ASCoR), > > > Kloveniersburgwal > > > > 48, 1012 CX Amsterdam. > > > > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > > > loet at leydesdorff.net ; > > > > http://www.leydesdorff.net/ > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: ASIS&T Special Interest Group on Metrics > > > > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of > > Eugene Garfield > > > > > Sent: Friday, March 03, 2006 6:37 PM > > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > > > > > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > > > and Herbert > > > > > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > > > > > 9 Jan 2006 > > > > > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > > Further to yesterday's posting, "Prestige is factored > > > into journal > > > > > ratings", here is another interesting and informative article > > > > > > > > > > FULL TEXT AVAILABLE AT : > > > > > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf > > > > > > > > > > email: {jbollen, marko, herbertv}@lanl.gov > > > > > > > > > > TITLE : Journal Status > > > > > > > > > > AUTHORS : Johan Bollen, Marko A. Rodriguez, and Herbert > > > > Van de Sompel > > > > > > > > > > SOURCE : arXiv:cs.GL/0601030 v1 9 Jan 2006 > > > > > > > > > > Abstract > > > > > The status of an actor in a social context is commonly > > defined in > > > > > terms of two factors: the total number of endorsements > > the actor > > > > > receives from other actors and the prestige of the > > > > endorsing actors. > > > > > These two factors indicate the distinction between > > popularity and > > > > > expert appreciation of the actor, respectively. We refer to > > > > the former > > > > > as popularity and to the latter as prestige. These notions of > > > > > popularity and prestige also apply to the domain of scholarly > > > > > assessment. The ISI Impact Factor (ISI IF) is defined > > as the mean > > > > > number of citations a journal receives over a 2 year > > > > period. By merely > > > > > counting the amount of citations and disregarding the > > > > prestige of the > > > > > citing journals, the ISI IF is a metric of popularity, not of > > > > > prestige. We demonstrate how a weighted version of > the popular > > > > > PageRank algorithm can be used to obtain a metric > that reflects > > > > > prestige. We contrast the rankings of journals according to > > > > their ISI > > > > > IF and their weighted PageRank, and we provide an > analysis that > > > > > reveals both significant overlaps and differences. > > > > > Furthermore, we introduce the Y-factor which is a simple > > > > combination > > > > > of both the ISI IF and the weighted PageRank, and > find that the > > > > > resulting journal rankings correspond well to a general > > > > understanding > > > > > of journal status. > > > > > > > > > > > > > > > ______________________________________________ > > > > > > > > > > > > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > > > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > > > > > > FULL TEXT AVAILABLE AT : > > > > > > > > > > > > > > > http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf OR > > > > > http://guide.labanimal.com/news/2006/060213/full/439770a.html > > > > > > > > > > > > > > > Philip Ball : p.ball at nature.com > > > > > www.philipball.com > > > > > > > > > > Title: Prestige is factored into journal ratings > > > > > > > > > > Author(s): Ball P > > > > > > > > > > Source: NATURE 439 (7078): 770-771 FEB 16 2006 > > > > > > > > > > Document Type: News Item Language: English > > > > > Cited References: 0 Times Cited: 0 > > > > > > > > > > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4 > > > > CRINAN ST, > > > > > LONDON > > > > > N1 9XW, ENGLAND > > > > > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA > > > > > > > > > > ISSN: 0028-0836 > > > > > > > > > > > > > > > > > > > From dgoodman at PRINCETON.EDU Tue Mar 7 20:54:21 2006 From: dgoodman at PRINCETON.EDU (David Goodman) Date: Tue, 7 Mar 2006 20:54:21 -0500 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: The distinction between review journals and primary journals has long been oversimplified. Some journals, even the best, have a mixture of both, such as Nature, or Science, or cerrtain biology. Some BMC titles fall into this cateogry as well, and for them, only the primary content is OA. Many primary journals place one review article in the front of each issue. This is sometimes written by invitation by a scientist of greater repute than their usual authors.It serves the purpose of attracting readers; it also serves the purpose of inflating the impact factor. Unlike purely review journals, these titles can not be derived from JCR formal criteria, such as the small number and great length of the articles. Many A&I services class any article with more than a cerrtain number of references as a review article, because such an article is usually comprehensive enough to serve the purposes of a review. I thus think the formal distinction unspecific, whether between journals or between individual articles, yet I recognize the need of it, and have nothing better to offer. Still, like all publication data, it should be used with the awareness of the ambiguities. Unfortunately, though a author may use it properly, the reader may not fully understand, and one cannot insert a caution of this length whenever the word is mentioned. What one can easily do is to link to the help pages from ISI: about review articles: http://jcr01.isiknowledge.com/JCR/help/h_sourcedata.htm#review_articles about review journals: http://jcr01.isiknowledge.com/JCR/help/h_using.htm (section on "Impact factor by article type.") Dr. David Goodman Associate Professor Palmer School of Library and Information Science Long Island University and formerly Princeton University Library dgoodman at liu.edu dgoodman at princeton.edu > > Dear Stephen, > > Journals are different in size, prestige, etc., but they are also > differently positioned in networks which differ in terms of the > densities of > the graphs. You are mainly interested in stratification within one > graph/discipline, that is, chemistry. For example, you are > interested in the > differences between review journals and article journals From dgoodman at PRINCETON.EDU Tue Mar 7 23:36:32 2006 From: dgoodman at PRINCETON.EDU (David Goodman) Date: Tue, 7 Mar 2006 23:36:32 -0500 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: Still on-list, there are several reasons for classifying journals. It can be for the practical purposes of arranging a library, or organizing information, and of interest mainly to those who do these things and wish to do it intelligently, or verify their intuition. This is quite a number of people, including A&I services. As a minimum,practical applications justify the more theoretical work. Clustering as a study in itself is interesting when it finds unexpected clusters, or those known only to specialists--such as the biopharmacology-plant science connection, or to help refine categories for specific uses, such as dividing up the BCMB subject heading. It rarely gives suprises that alert practical people do not already know. Clustering is part of a bibliometric analysis, which has many subsequent directions. A related subject is the history of publishing and of journals, where bibliometrics is one of the methods of analysis. But most of what I think we are discussing is clustering for the purpose of ranking journals; its obviously a necessary preliminary. where it is the first step (usually w do not want to rank all the journals in the world, as the early BLL studies did.) Why is this of practical interest? Obviously, to clarify what journals are of value, and should therefore be acquired. I am not sure of the relevance of global analysis here, because one is usually acquiring journals for some specific institution. In such cases the global data is of use mainly in providing a background to compare with the local measures, and to spot idiosncratic needs. This is not merely an exercise in statistics, and citation analysis is but one of the measures. In practice, global citation analysis has been often used, being the only objective measure at hand, except for price and size. As other measures develop, we need valid ways of incorporating them. So far, for scholarly journals in many subjects, none have been shown more useful than IF alone, employed intelligently. We might be talking about journals so high ranking that they ought to be acquired whether there is an immediate need or not? This used to be necessary to accommodate future diversification when financially possible, but with e-journals is irrelevant, because the complete runs can be obtained when they become necessary. Why do we care about what journals to acquire? Why shouldn't we simply obtain access to them all? The main reason is because this is not yet practical, and we do have to provide for the current pre-OA environoment. (It can also be argued that a selected list is of use to beginners--but that is another discussion and possibly another audience.) Dr. David Goodman Associate Professor Palmer School of Library and Information Science Long Island University and formerly Princeton University Library dgoodman at liu.edu dgoodman at princeton.edu [I omit the sequence we have all read previously] From loet at LEYDESDORFF.NET Wed Mar 8 01:39:08 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Wed, 8 Mar 2006 07:39:08 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: Dear David, The problem is that there are different hierarchies intermingled because of the differentiation in the cognitive dimension. Some hierarchies may stand orthogonal on others. Thus, the problem is not with _Science_ and _Nature_ at the top, but in the intermediate ranges. There is no single hierarchy because there are two effects interacting: status in the stratification and density in the relevant environment (graph or cluster). Sometimes, there is almost empty space between the clusters and the cluster structures are discipline-specific. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of David Goodman > Sent: Wednesday, March 08, 2006 5:37 AM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Still on-list, there are several reasons for classifying > journals. It can be for the practical purposes of arranging > a library, or organizing information, and of interest mainly > to those who do these things and wish to do it intelligently, > or verify their intuition. This is quite a number of people, > including A&I services. > As a minimum,practical applications justify the more theoretical work. > > Clustering as a study in itself is interesting when it finds > unexpected clusters, or those known only to specialists--such > as the biopharmacology-plant science connection, or to help > refine categories for specific uses, such as dividing > up the BCMB subject heading. It rarely gives suprises that > alert practical people do not already know. > > Clustering is part of a bibliometric analysis, which has > many subsequent directions. A related subject is the history > of publishing and of journals, where bibliometrics is one of > the methods of analysis. > > But most of what I think we are discussing is clustering for > the purpose of ranking journals; its obviously a necessary > preliminary. where it is the first step (usually w do not > want to rank all the journals in the world, as the early BLL > studies did.) > > Why is this of practical interest? Obviously, to clarify what > journals are of value, and should therefore be acquired. I am > not sure of the relevance of global analysis here, because > one is usually acquiring journals for some specific > institution. In such cases the global data is of use mainly > in providing a background to compare with the local measures, > and to spot idiosncratic needs. > > This is not merely an exercise in statistics, and citation > analysis is but one of the measures. In practice, global > citation analysis has been often used, being the only > objective measure at hand, except for price and size. As > other measures develop, we need valid ways of incorporating > them. So far, for scholarly journals in many subjects, none > have been shown more useful than IF alone, employed intelligently. > > We might be talking about journals so high ranking that they > ought to be acquired whether there is an immediate need or > not? This used to be necessary to accommodate future > diversification when financially possible, but with > e-journals is irrelevant, because the complete runs can be > obtained when they become necessary. > > Why do we care about what journals to acquire? Why shouldn't > we simply obtain access to them all? The main reason is > because this is not yet practical, and we do have to provide > for the current pre-OA environoment. (It can also be argued > that a selected list is of use to beginners--but that is > another discussion and possibly another > audience.) > > Dr. David Goodman > Associate Professor > Palmer School of Library and Information Science Long Island > University and formerly Princeton University Library > > dgoodman at liu.edu > dgoodman at princeton.edu > > [I omit the sequence we have all read previously] > From loet at LEYDESDORFF.NET Wed Mar 8 01:47:01 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Wed, 8 Mar 2006 07:47:01 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: > PS I use ISI subject categories and have found them very > good. But then I am used to using bad classification systems > like LC and the DDC. I am happy for you! However, this was not my question. I was just wondering about the methodological legitimation of these categories. Can one use something which one does not understand -- and nobody seems to understand them -- as a basis for research and evaluation decisions? One knows that outcomes of, for example, bibliometric assessments are heavily dependent on this type of delineations. Perhaps, I am the more old-fashioned one of the two of us. :-) Best, Loet From havemanf at CMS.HU-BERLIN.DE Wed Mar 8 10:31:16 2006 From: havemanf at CMS.HU-BERLIN.DE (Frank Havemann) Date: Wed, 8 Mar 2006 16:31:16 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: The main problem I have with this paper is: The weighted "PageRank" algorithm was proposed already in 1978 for citation networks of journals by Nancy L. Geller (see below). The difference of Geller and PageRank is only that in the case of journals normally a damping factor is not needed (lambda = 1). In the paper by Bollen et al. I have not found any hint for which reason they need damping and how different damping factors influence their results. @Article{Geller1978citation, author = {Nancy L. Geller}, title = {{On the citation influence methodology of Pinski and Narin}}, journal = {Information Processing \& Management}, year = {1978}, volume = {14}, number = {2}, pages = {93--95}, } Frank Havemann *************************** Dr. Frank Havemann Department of Library and Information Science Humboldt University Dorotheenstr. 26 D-10099 Berlin Germany tel.: (0049) (030) 2093 4228 http://www.ib.hu-berlin.de/inf/havemann.html From notsjb at LSU.EDU Wed Mar 8 11:18:35 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Wed, 8 Mar 2006 10:18:35 -0600 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: I really do not understand what you mean by "methodological legitimation" and why these categories need to be legitimized. I look at the titles in the categories, and the words of the titles are enough to legitimize them. I only suspect problems when I am plagued by outliers and interaction effects. When I first started to analyze "chemistry" journals, I once told a colleague that the entire thing would be simple, if I could only figure out what a "chemistry" journal was, and he laughed at me. Now he no longer laughs at me when I use terms such as "chemistryness" to indicate that the journals are only an outward manifestation in the material world of the Platonic Idea of "chemistry." Every time I drive my car or use a computer I use something that I don't understand. I still think that my computer consists of little mice running on treadmills. I use a lot of statistics, whose mathematical bases are entirely beyond me. It is just enough for me that these things work. The economic term, I think, is "specialization." SB Loet Leydesdorff @listserv.utk.edu> on 03/08/2006 12:47:01 AM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at listserv.utk.edu cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 > PS I use ISI subject categories and have found them very > good. But then I am used to using bad classification systems > like LC and the DDC. I am happy for you! However, this was not my question. I was just wondering about the methodological legitimation of these categories. Can one use something which one does not understand -- and nobody seems to understand them -- as a basis for research and evaluation decisions? One knows that outcomes of, for example, bibliometric assessments are heavily dependent on this type of delineations. Perhaps, I am the more old-fashioned one of the two of us. :-) Best, Loet From notsjb at LSU.EDU Wed Mar 8 11:43:24 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Wed, 8 Mar 2006 10:43:24 -0600 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: Stephen J Bensman 03/08/2006 10:39 AM To: dgoodman at Princeton.EDU cc: Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 (Document link: Stephen J Bensman) Good point well made, David. Even the set of review journals is fuzzy. However, I think that Garfield beat you to this insight some 25 years ago. You might want to peruse his articles posted at the following URLs: http://www.garfield.library.upenn.edu/essays/v5p695y1981-82.pdf http://www.garfield.library.upenn.edu/essays/v10p113y1987.pdf In these he states that the word "review" is one of the more ambiguous terms in science and cites research classifying review journals into eight different types. Mark Twain once quipped that the trouble with the ancients is that they stole our best ideas. SB David Goodman @LISTSERV.UTK.EDU> on 03/07/2006 07:54:21 PM Please respond to dgoodman at Princeton.EDU Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 The distinction between review journals and primary journals has long been oversimplified. Some journals, even the best, have a mixture of both, such as Nature, or Science, or cerrtain biology. Some BMC titles fall into this cateogry as well, and for them, only the primary content is OA. Many primary journals place one review article in the front of each issue. This is sometimes written by invitation by a scientist of greater repute than their usual authors.It serves the purpose of attracting readers; it also serves the purpose of inflating the impact factor. Unlike purely review journals, these titles can not be derived from JCR formal criteria, such as the small number and great length of the articles. Many A&I services class any article with more than a cerrtain number of references as a review article, because such an article is usually comprehensive enough to serve the purposes of a review. I thus think the formal distinction unspecific, whether between journals or between individual articles, yet I recognize the need of it, and have nothing better to offer. Still, like all publication data, it should be used with the awareness of the ambiguities. Unfortunately, though a author may use it properly, the reader may not fully understand, and one cannot insert a caution of this length whenever the word is mentioned. What one can easily do is to link to the help pages from ISI: about review articles: http://jcr01.isiknowledge.com/JCR/help/h_sourcedata.htm#review_articles about review journals: http://jcr01.isiknowledge.com/JCR/help/h_using.htm (section on "Impact factor by article type.") Dr. David Goodman Associate Professor Palmer School of Library and Information Science Long Island University and formerly Princeton University Library dgoodman at liu.edu dgoodman at princeton.edu > > Dear Stephen, > > Journals are different in size, prestige, etc., but they are also > differently positioned in networks which differ in terms of the > densities of > the graphs. You are mainly interested in stratification within one > graph/discipline, that is, chemistry. For example, you are > interested in the > differences between review journals and article journals From loet at LEYDESDORFF.NET Wed Mar 8 13:35:07 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Wed, 8 Mar 2006 19:35:07 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: I would happily leave it to you if it were only "chemistry." Best, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman > Sent: Wednesday, March 08, 2006 5:19 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > I really do not understand what you mean by "methodological > legitimation" > and why these categories need to be legitimized. I look at > the titles in the categories, and the words of the titles are > enough to legitimize them. > I only suspect problems when I am plagued by outliers and > interaction effects. > > When I first started to analyze "chemistry" journals, I once > told a colleague that the entire thing would be simple, if I > could only figure out what a "chemistry" journal was, and he > laughed at me. Now he no longer laughs at me when I use > terms such as "chemistryness" to indicate that the journals > are only an outward manifestation in the material world of > the Platonic Idea of "chemistry." > > Every time I drive my car or use a computer I use something > that I don't understand. I still think that my computer > consists of little mice running on treadmills. I use a lot > of statistics, whose mathematical bases are entirely beyond > me. It is just enough for me that these things work. The > economic term, I think, is "specialization." > > SB > > > > > Loet Leydesdorff @listserv.utk.edu> on > 03/08/2006 > 12:47:01 AM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at listserv.utk.edu > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > Rodriguez, and Herbert > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > > PS I use ISI subject categories and have found them very > good. But > > then I am used to using bad classification systems like LC and the > > DDC. > > I am happy for you! However, this was not my question. I was > just wondering about the methodological legitimation of these > categories. Can one use something which one does not > understand -- and nobody seems to understand them -- as a > basis for research and evaluation decisions? One knows that > outcomes of, for example, bibliometric assessments are > heavily dependent on this type of delineations. > > Perhaps, I am the more old-fashioned one of the two of us. :-) > > Best, Loet > From garfield at CODEX.CIS.UPENN.EDU Wed Mar 8 13:49:52 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Wed, 8 Mar 2006 13:49:52 -0500 Subject: Diamond AM "Edwin Mansfield's contributions to the economics of technology " Research Policy 32(9):1607-1617, October 2003 Message-ID: Arthur Diamond : adiamond at mail.unomaha.edu Title: Edwin Mansfield's contributions to the economics of technology Author(s): Diamond AM Source: RESEARCH POLICY 32 (9): 1607-1617 OCT 2003 Document Type: Article Language: English Cited References: 55 Times Cited: 0 Abstract: Edwin Mansfield's contributions to the economics of technology are summarized from the early 1960s through his death in 1997. Mansfield's methodology is discussed, as are his contributions on: the diffusion of technical innovation, the effect of firm size on innovation, the role of academic and basic research in increasing innovation and productivity, international technology transfer and the inaccuracy of technological forecasts. The economics profession's evaluation of the relative importance of Mansfield's work is presented, using as evidence citation counts of his works collected from the Social Science Citation Index (SSCI). Identified as among Mansfield's most important contributions are his work on the importance of academic research for industrial innovations, his empirical estimation of the rates of diffusion of different innovations, and his estimation of the private and social returns from investments in industrial innovations. Finally, we present Mansfield's advice on the future of the economics of technology. (C) 2003 Elsevier Science B.V All rights reserved. Addresses: Diamond AM (reprint author), Univ Nebraska, Dept Econ, Omaha, NE 68182 USA Univ Nebraska, Dept Econ, Omaha, NE 68182 USA Publisher: ELSEVIER SCIENCE BV, PO BOX 211, 1000 AE AMSTERDAM, NETHERLANDS Subject Category: MANAGEMENT; PLANNING AND DEVELOPMENT IDS Number: 735KC ISSN: 0048-7333 CITED REFERENCES : FOST ASS INC FIN REP SURV NET RAT 2 : 1978 ARROW KJ Preface: Edwin Mansfield's research on technology and innovation INTERNATIONAL JOURNAL OF TECHNOLOGY MANAGEMENT 19 : 1 2000 BEARDSLEY G NOTE ON ACCURACY OF INDUSTRIAL FORECASTS OF PROFITABILITY OF NEW PRODUCTS AND PROCESSES JOURNAL OF BUSINESS 51 : 127 1978 DIAMOND AM KNOWL POLICY 9 : 6 1996 DOSI G SOURCES, PROCEDURES, AND MICROECONOMIC EFFECTS OF INNOVATION JOURNAL OF ECONOMIC LITERATURE 26 : 1120 1988 GRANSTRAND O EC TECHNOLOGY : 1 1994 GRILICHES Z HYBRID CORN - AN EXPLORATION IN THE ECONOMICS OF TECHNOLOGICAL-CHANGE ECONOMETRICA 25 : 501 1957 GRILICHES Z PATENT STATISTICS AS ECONOMIC INDICATORS - A SURVEY JOURNAL OF ECONOMIC LITERATURE 28 : 1661 1990 KEALEY T EC LAWS SCI RES : 1996 KEALEY T END GOVT SCI FUNDING : 1999 KONDO EK The effect of patent protection on foreign direct investment JOURNAL OF WORLD TRADE 29 : 97 1995 KONDO EK THESIS HARVARD U : 1995 LANE N FEDERAL PRIORITIES O : 1998 MANSFIELD E INDUSTRIAL R-AND-D IN JAPAN AND THE UNITED-STATES - A COMPARATIVE-STUDY AMERICAN ECONOMIC REVIEW 78 : 223 1988 MANSFIELD E TECHNOLOGICAL-CHANGE AND MARKET-STRUCTURE - AN EMPIRICAL-STUDY AMERICAN ECONOMIC REVIEW 73 : 205 1983 MANSFIELD E BASIC RESEARCH AND PRODUCTIVITY INCREASE IN MANUFACTURING AMERICAN ECONOMIC REVIEW 70 : 863 1980 MANSFIELD E TECHNOLOGICAL CHANGES - STIMULI, CONSTRAINTS, RETURNS - RATES OF RETURN FROM INDUSTRIAL-RESEARCH AND DEVELOPMENT AMERICAN ECONOMIC REVIEW 55 : 310 1965 MANSFIELD E ENTRY, GIBRAT LAW, INNOVATION, AND THE GROWTH OF FIRMS AMERICAN ECONOMIC REVIEW 52 : 1023 1962 MANSFIELD E EC TECHNICAL CHANGE : 1993 MANSFIELD E EC TECHNOLOGICAL CHA : 1968 MANSFIELD E IMITATION COSTS AND PATENTS - AN EMPIRICAL-STUDY ECONOMIC JOURNAL 91 : 907 1981 MANSFIELD E TECHNICAL CHANGE AND THE RATE OF IMITATION ECONOMETRICA 29 : 741 1961 MANSFIELD E OVERSEAS RESEARCH AND DEVELOPMENT BY UNITED-STATES BASED FIRMS ECONOMICA 46 : 187 1979 MANSFIELD E GLOBAL DIMENSIONS IN : 107 1993 MANSFIELD E HOW ECONOMISTS SEE R-AND-D HARVARD BUSINESS REVIEW 59 : 98 1981 MANSFIELD E IND RES TECHNOLOGICA : 1968 MANSFIELD E INNOVATION TECHNOLOG 2 : 1995 MANSFIELD E INNOVATION TECHNOLOG 1 : 1995 MANSFIELD E ORGANIZATIONAL AND STRATEGIC FACTORS ASSOCIATED WITH PROBABILITIES OF SUCCESS IN INDUSTRIAL R AND D JOURNAL OF BUSINESS 48 : 179 1975 MANSFIELD E The economics of growth and technical change: Technologies, nations, agents - Silverberg,G, Soete,L JOURNAL OF ECONOMIC LITERATURE 34 : 179 1996 MANSFIELD E HOW RAPIDLY DOES NEW INDUSTRIAL-TECHNOLOGY LEAK OUT JOURNAL OF INDUSTRIAL ECONOMICS 34 : 217 1985 MANSFIELD E INDUSTRIAL-RESEARCH AND DEVELOPMENT EXPENDITURES DETERMINANTS, PROSPECTS, AND RELATION TO SIZE OF FIRM AND INVENTIVE OUTPUT JOURNAL OF POLITICAL ECONOMY 72 : 319 1964 MANSFIELD E SIZE OF FIRM, MARKET-STRUCTURE, AND INNOVATION JOURNAL OF POLITICAL ECONOMY 71 : 556 1963 MANSFIELD E THE SPEED AND COST OF INDUSTRIAL-INNOVATION IN JAPAN AND THE UNITED-STATES - EXTERNAL VS INTERNAL TECHNOLOGY MANAGEMENT SCIENCE 34 : 1157 1988 MANSFIELD E PATENTS AND INNOVATION - AN EMPIRICAL-STUDY MANAGEMENT SCIENCE 32 : 173 1986 MANSFIELD E OBITUARY : 1997 MANSFIELD E PRODUCTION APPL NEW : 1977 MANSFIELD E SOCIAL AND PRIVATE RATES OF RETURN FROM INDUSTRIAL INNOVATIONS QUARTERLY JOURNAL OF ECONOMICS 91 : 221 1977 MANSFIELD E THE SPEED OF RESPONSE OF FIRMS TO NEW TECHNIQUES QUARTERLY JOURNAL OF ECONOMICS 77 : 290 1963 MANSFIELD E RES INNOVATION MODER : 1971 MANSFIELD E ACADEMIC RESEARCH AND INDUSTRIAL-INNOVATION - A FURTHER NOTE RESEARCH POLICY 21 : 295 1992 MANSFIELD E ACADEMIC RESEARCH AND INDUSTRIAL-INNOVATION RESEARCH POLICY 20 : 1 1991 MANSFIELD E COMPOSITION OF R-AND-D EXPENDITURES - RELATIONSHIP TO SIZE OF FIRM, CONCENTRATION, AND INNOVATIVE OUTPUT REVIEW OF ECONOMICS AND STATISTICS 63 : 610 1981 MANSFIELD E FOREIGN-TRADE AND UNITED-STATES RESEARCH AND DEVELOPMENT REVIEW OF ECONOMICS AND STATISTICS 61 : 49 1979 MANSFIELD E INTRAFIRM RATES OF DIFFUSION OF AN INNOVATION REVIEW OF ECONOMICS AND STATISTICS 45 : 348 1963 MANSFIELD E INDUSTRIAL-INNOVATION IN JAPAN AND THE UNITED-STATES SCIENCE 241 : 1769 1988 MANSFIELD E CONTRIBUTION OF R AND D TO ECONOMIC GROWTH IN UNITED-STATES SCIENCE 175 : 477 1972 MANSFIELD E TECHNOLOGICAL CHANGE : 77 1986 MANSFIELD E TECHNOLOGICAL CHANGE : 1971 MANSFIELD E TECHNOLOGY TRANSFER : 1982 MEDOFF MH AM ECON 40 : 46 1996 NADIRI MI SOME APPROACHES TO THEORY AND MEASUREMENT OF TOTAL FACTOR PRODUCTIVITY - SURVEY JOURNAL OF ECONOMIC LITERATURE 8 : 1137 1970 ROBERT R NSF7622519 NATH ASS : 1978 ROSENBERG N INSIDE BLACK BOX TEC : 1982 SCHERER FM R-AND-D AND DECLINING PRODUCTIVITY GROWTH AMERICAN ECONOMIC REVIEW 73 : 215 1983 From notsjb at LSU.EDU Wed Mar 8 15:17:30 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Wed, 8 Mar 2006 14:17:30 -0600 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: Loet, I just finished a tutorial by Thomson Scientific on use of the online JCR. During the course of this tutorial, I found out how ISI assigns journals to its subject categories. It is so delicious that I cannot resist telling you what I learned. They have for each subject category a little blurb describing what that category is and what it covers. It is part of the category information. An ISI person takes a journal, looks at 2 or 3 issues, and then SUBJECTIVELY--I REPEAT, SUBJECTIVELY--assigns the journal to a subject category on the basis of those category descriptions. It is so traditional, librarian, and Keynesian that your name immediately popped into my mind together with a howl of laughter. In terms on how these things have been done for centuries, it is definitely legitmized. SB Loet Leydesdorff @listserv.utk.edu> on 03/08/2006 12:47:01 AM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at listserv.utk.edu cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 > PS I use ISI subject categories and have found them very > good. But then I am used to using bad classification systems > like LC and the DDC. I am happy for you! However, this was not my question. I was just wondering about the methodological legitimation of these categories. Can one use something which one does not understand -- and nobody seems to understand them -- as a basis for research and evaluation decisions? One knows that outcomes of, for example, bibliometric assessments are heavily dependent on this type of delineations. Perhaps, I am the more old-fashioned one of the two of us. :-) Best, Loet From loet at LEYDESDORFF.NET Wed Mar 8 15:46:15 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Wed, 8 Mar 2006 21:46:15 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: Great! Thanks a lot. I am happy to hear that you can laugh about it. :-) Best, Loet > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman > Sent: Wednesday, March 08, 2006 9:17 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Loet, > I just finished a tutorial by Thomson Scientific on use of > the online JCR. > During the course of this tutorial, I found out how ISI > assigns journals to its subject categories. It is so > delicious that I cannot resist telling you what I learned. > > They have for each subject category a little blurb describing > what that category is and what it covers. It is part of the > category information. > An ISI person takes a journal, looks at 2 or 3 issues, and > then SUBJECTIVELY--I REPEAT, SUBJECTIVELY--assigns the > journal to a subject category on the basis of those category > descriptions. It is so traditional, librarian, and Keynesian > that your name immediately popped into my mind together with > a howl of laughter. > > In terms on how these things have been done for centuries, it > is definitely legitmized. > > SB > > > > > Loet Leydesdorff @listserv.utk.edu> on > 03/08/2006 > 12:47:01 AM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at listserv.utk.edu > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > Rodriguez, and Herbert > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > > PS I use ISI subject categories and have found them very > good. But > > then I am used to using bad classification systems like LC and the > > DDC. > > I am happy for you! However, this was not my question. I was > just wondering about the methodological legitimation of these > categories. Can one use something which one does not > understand -- and nobody seems to understand them -- as a > basis for research and evaluation decisions? One knows that > outcomes of, for example, bibliometric assessments are > heavily dependent on this type of delineations. > > Perhaps, I am the more old-fashioned one of the two of us. :-) > > Best, Loet > From notsjb at LSU.EDU Wed Mar 8 15:56:25 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Wed, 8 Mar 2006 14:56:25 -0600 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: What's not to laugh about it? There probably is no better way to do it. SB Loet Leydesdorff @LISTSERV.UTK.EDU> on 03/08/2006 02:46:15 PM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Great! Thanks a lot. I am happy to hear that you can laugh about it. :-) Best, Loet > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman > Sent: Wednesday, March 08, 2006 9:17 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Loet, > I just finished a tutorial by Thomson Scientific on use of > the online JCR. > During the course of this tutorial, I found out how ISI > assigns journals to its subject categories. It is so > delicious that I cannot resist telling you what I learned. > > They have for each subject category a little blurb describing > what that category is and what it covers. It is part of the > category information. > An ISI person takes a journal, looks at 2 or 3 issues, and > then SUBJECTIVELY--I REPEAT, SUBJECTIVELY--assigns the > journal to a subject category on the basis of those category > descriptions. It is so traditional, librarian, and Keynesian > that your name immediately popped into my mind together with > a howl of laughter. > > In terms on how these things have been done for centuries, it > is definitely legitmized. > > SB > > > > > Loet Leydesdorff @listserv.utk.edu> on > 03/08/2006 > 12:47:01 AM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at listserv.utk.edu > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. > Rodriguez, and Herbert > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 > 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > > PS I use ISI subject categories and have found them very > good. But > > then I am used to using bad classification systems like LC and the > > DDC. > > I am happy for you! However, this was not my question. I was > just wondering about the methodological legitimation of these > categories. Can one use something which one does not > understand -- and nobody seems to understand them -- as a > basis for research and evaluation decisions? One knows that > outcomes of, for example, bibliometric assessments are > heavily dependent on this type of delineations. > > Perhaps, I am the more old-fashioned one of the two of us. :-) > > Best, Loet > From loet at LEYDESDORFF.NET Thu Mar 9 01:25:45 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Thu, 9 Mar 2006 07:25:45 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: > What's not to laugh about it? There probably is no better > way to do it. > > SB Let me then repeat the problem: comparisons in terms of impact factors, etc., are valid only within cognitive domains with common citation and publication practices. In other words, citation graphs among journals have different densities and this affects the impact factors in the corresponding domains. For example, impact factors of immunology journals are much higher than impact factors of toxicology journals. The delineation of the sets in which one can compare thus matters. We know that this delineation cannot be perfect, but it matters how good they are. Increasingly evaluation commission and scientometric researchers seem to assume that the ISI subject categories are valid delineation of domains within which one can make comparisons. The article by Bollen et al. was a point in case. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ From george at LIBRARY.CALTECH.EDU Thu Mar 9 01:52:30 2006 From: george at LIBRARY.CALTECH.EDU (George Porter) Date: Wed, 8 Mar 2006 22:52:30 -0800 Subject: Dana Roth on citation indexing Message-ID: My colleague, Dana Roth, has published an overview of the variety of fee and free citation searching services which have sprung up over the years. Dana L. Roth. (2005) The emergence of competitors to the Science Citation Index and the Web of Science. Current Science 89(9):1531-6 (10 November 2005) His contribution was part of a "Special Section on 50 Years of Citation Indexing". Current Science , just as all of the other publications of the Indian Academy of Sciences , is an Open Access journal. George S. Porter Sherman Fairchild Library of Engineering & Applied Science California Institute of Technology Mail Code 1-43, Pasadena, CA 91125-4300 Telephone (626) 395-3409 Fax (626) 431-2681 http://library.caltech.edu contributor http://stlq.info | http://www.earlham.edu/~peters/fos/fosblog.html From loet at LEYDESDORFF.NET Thu Mar 9 03:51:27 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Thu, 9 Mar 2006 09:51:27 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: <200603081631.16234.havemanf@cms.hu-berlin.de> Message-ID: Dear Frank: Nancy Geller (1974) elegantly formulates the problem which is also valid for IFs: "We also observe that the Markov Chain context explains why caution is necessary in comparing influence weights which arise from separate aggregates of journals (etc). Comparing one physics journal that has influence weight four among physics journals to another physics journal which has influence weight three among the same physics journals is not the same as comparing a physics journal which has influence weight four among physics journals with a chemistry journal which has influence weight three among chemistry journals. One explanation of this is that an influence weight is a function of the long-run probability of being cited within the aggregate itself. Thus a journal's influence weight depends on which journals it is aggregated with, and different aggregations will ordinarily lead to different influence weights for the same journal." In other words, the zero's in the citation matrix make the system nearly decomposable. This is reinforced by ISI's decision to lump all ones under the category "All others". This makes the matrix even more sparse. Thus, one first needs the decomposition before one can make comparisons. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Frank Havemann > Sent: Wednesday, March 08, 2006 4:31 PM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, > and Herbert Van de Sompel "Journal Status" > arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > The main problem I have with this paper is: > > The weighted "PageRank" algorithm was proposed already in > 1978 for citation networks of journals by Nancy L. Geller (see below). > > The difference of Geller and PageRank is only that in the > case of journals normally a damping factor is not needed (lambda = 1). > > In the paper by Bollen et al. I have not found any hint for > which reason they need damping and how different damping > factors influence their results. > > @Article{Geller1978citation, > author = {Nancy L. Geller}, > title = {{On the citation influence methodology of Pinski and > Narin}}, journal = {Information Processing \& Management}, > year = {1978}, volume = {14}, number = {2}, pages = {93--95}, } > > Frank Havemann > > > *************************** > Dr. Frank Havemann > Department of Library and Information Science Humboldt > University Dorotheenstr. 26 > D-10099 Berlin > Germany > > tel.: (0049) (030) 2093 4228 > http://www.ib.hu-berlin.de/inf/havemann.html > From notsjb at LSU.EDU Thu Mar 9 09:28:28 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Thu, 9 Mar 2006 08:28:28 -0600 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: Loet, That is pretty basic, but the problem is that due to Bradford's Law and Garfield's Law definition of such sets is impossible. You are always going to have exogenous variables if you use citations. If you going to use citations in evaluations, they must be used together with other variables--the best being experta ratings if such are available. Then you can check for extreme outliers indicating sources of distortion. The evaluation must be specific to those scientists being evaluated. It is not possible define mathematically sets universally applicable. The one thing that really bothers me about European research is that they seem to assume that citations are valid measures of quality. It then concentrates on find some mathematical technique supposedly capable of measuring quality. This research seems woefully short of studies of the opinions of actual scientists as well as the institutional and social bases of citations. It seems to boil down to fascination with new gimmickry--latest being the present fad with the Hirsch index, Compared with the work done by the American Council on Education and the US National Research Council it is quite crude--even the vaunted British RAE. What the Americans have found is that no matter how carefully you do it, you always crap it up somehow. SB Loet Leydesdorff @LISTSERV.UTK.EDU> on 03/09/2006 12:25:45 AM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 > What's not to laugh about it? There probably is no better > way to do it. > > SB Let me then repeat the problem: comparisons in terms of impact factors, etc., are valid only within cognitive domains with common citation and publication practices. In other words, citation graphs among journals have different densities and this affects the impact factors in the corresponding domains. For example, impact factors of immunology journals are much higher than impact factors of toxicology journals. The delineation of the sets in which one can compare thus matters. We know that this delineation cannot be perfect, but it matters how good they are. Increasingly evaluation commission and scientometric researchers seem to assume that the ISI subject categories are valid delineation of domains within which one can make comparisons. The article by Bollen et al. was a point in case. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ From loet at LEYDESDORFF.NET Thu Mar 9 10:31:47 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Thu, 9 Mar 2006 16:31:47 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: Dear Stephen, In your categorization, I am fullheartedly on the American side. I would be very hesitant to use the word quality in this context. My interest is in scientific communication, its structures, and its development. With best wishes, Loet ----- Original Message ----- From: "Stephen J Bensman" To: Sent: Thursday, March 09, 2006 3:28 PM Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Loet, > That is pretty basic, but the problem is that due to Bradford's Law and > Garfield's Law definition of such sets is impossible. You are always > going > to have exogenous variables if you use citations. If you going to use > citations in evaluations, they must be used together with other > variables--the best being experta ratings if such are available. Then you > can check for extreme outliers indicating sources of distortion. The > evaluation must be specific to those scientists being evaluated. It is > not possible define mathematically sets universally applicable. > > The one thing that really bothers me about European research is that they > seem to assume that citations are valid measures of quality. It then > concentrates on find some mathematical technique supposedly capable of > measuring quality. This research seems woefully short of studies of the > opinions of actual scientists as well as the institutional and social > bases > of citations. It seems to boil down to fascination with new > gimmickry--latest being the present fad with the Hirsch index, Compared > with the work done by the American Council on Education and the US > National > Research Council it is quite crude--even the vaunted British RAE. What > the > Americans have found is that no matter how carefully you do it, you always > crap it up somehow. > > SB > > > > > Loet Leydesdorff @LISTSERV.UTK.EDU> on 03/09/2006 > 12:25:45 AM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > >> What's not to laugh about it? There probably is no better >> way to do it. >> >> SB > > Let me then repeat the problem: comparisons in terms of impact factors, > etc., are valid only within cognitive domains with common citation and > publication practices. In other words, citation graphs among journals have > different densities and this affects the impact factors in the > corresponding > domains. For example, impact factors of immunology journals are much > higher > than impact factors of toxicology journals. > > The delineation of the sets in which one can compare thus matters. We know > that this delineation cannot be perfect, but it matters how good they are. > Increasingly evaluation commission and scientometric researchers seem to > assume that the ISI subject categories are valid delineation of domains > within which one can make comparisons. The article by Bollen et al. was a > point in case. > > With best wishes, > > > Loet > ________________________________ > Loet Leydesdorff > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal 48, 1012 CX Amsterdam. > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; http://www.leydesdorff.net/ From kboyack at SANDIA.GOV Thu Mar 9 10:31:26 2006 From: kboyack at SANDIA.GOV (Boyack, Kevin W) Date: Thu, 9 Mar 2006 08:31:26 -0700 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: Stephen, Can you point me to any references that would quantify your statement: "What the Americans have found is that no matter how carefully you do it, you always crap it up somehow." I would love to read the about the lessons learned. Thanks, Kevin -----Original Message----- From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman Sent: Thursday, March 09, 2006 7:28 AM To: SIGMETRICS at LISTSERV.UTK.EDU Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Loet, That is pretty basic, but the problem is that due to Bradford's Law and Garfield's Law definition of such sets is impossible. You are always going to have exogenous variables if you use citations. If you going to use citations in evaluations, they must be used together with other variables--the best being experta ratings if such are available. Then you can check for extreme outliers indicating sources of distortion. The evaluation must be specific to those scientists being evaluated. It is not possible define mathematically sets universally applicable. The one thing that really bothers me about European research is that they seem to assume that citations are valid measures of quality. It then concentrates on find some mathematical technique supposedly capable of measuring quality. This research seems woefully short of studies of the opinions of actual scientists as well as the institutional and social bases of citations. It seems to boil down to fascination with new gimmickry--latest being the present fad with the Hirsch index, Compared with the work done by the American Council on Education and the US National Research Council it is quite crude--even the vaunted British RAE. What the Americans have found is that no matter how carefully you do it, you always crap it up somehow. SB Loet Leydesdorff @LISTSERV.UTK.EDU> on 03/09/2006 12:25:45 AM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 > What's not to laugh about it? There probably is no better way to do > it. > > SB Let me then repeat the problem: comparisons in terms of impact factors, etc., are valid only within cognitive domains with common citation and publication practices. In other words, citation graphs among journals have different densities and this affects the impact factors in the corresponding domains. For example, impact factors of immunology journals are much higher than impact factors of toxicology journals. The delineation of the sets in which one can compare thus matters. We know that this delineation cannot be perfect, but it matters how good they are. Increasingly evaluation commission and scientometric researchers seem to assume that the ISI subject categories are valid delineation of domains within which one can make comparisons. The article by Bollen et al. was a point in case. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ From bernies at UILLINOIS.EDU Thu Mar 9 10:42:02 2006 From: bernies at UILLINOIS.EDU (Sloan, Bernie) Date: Thu, 9 Mar 2006 09:42:02 -0600 Subject: Current Science: 50 Years of Citation Indexing Message-ID: Special Section: 50 Years of Citation Indexing. Current Science, 89(9), 1502-1554. November 10, 2005. http://www.ias.ac.in/currsci/nov102005/contents.htm Bernie Sloan Senior Information Systems Consultant Consortium of Academic & Research Libraries in Illinois 616 E. Green Street, Suite 213 Champaign, IL 61820-5752 Phone: (217) 333-4895 Fax: (217) 265-0454 E-mail: bernies at uillinois.edu From notsjb at LSU.EDU Thu Mar 9 12:17:38 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Thu, 9 Mar 2006 11:17:38 -0600 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: The literature is full of these screw ups. Perhaps the funniest example is what happened with music, which was evaluated on the same basis as physics. However, musicians play music and do not write papers. Therefore, the Juillard School of Music was rated very low. It refused to undergo another such humiliating experience. Probably the most important example causing the biggest foul up involved the biosciences. This involved an error under discussion here. Ratings were traditionally done on an organizational basis. However, biosciences are organized differently at different institutions. Some have med schools, some do not, Some have agricultural schools, some do not. The organizational basis caused an inability to rate comparable sets at different universities, and this severely affected LSU. In 1981 LSU put two small departments in the College of Basic Sciences up for ratings, and these were creamed. But in 1993 it was decided base the ratings on subject categories instead organizational units. As a result, LSU put up for ratings all its bioscientists not only in the College of Basic Sciences but all at Vet Med and in the College of Agriculture on the Baton Rouge campus. Since scientific significance is a function of size, LSU jumped in the rankings in a probabilistically impossible fashion. LSU had its med schools at New Orleans and Shreveport rated separately, and if these had been thrown into the mix, its rankings would have been even higher. So it seems that from 1910 through 1993 all bioscience ratings were incorrect due to comparing incomparable sets. Hope you found this bit of person experience interesting. SB "Boyack, Kevin W" @LISTSERV.UTK.EDU> on 03/09/2006 09:31:26 AM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Stephen, Can you point me to any references that would quantify your statement: "What the Americans have found is that no matter how carefully you do it, you always crap it up somehow." I would love to read the about the lessons learned. Thanks, Kevin -----Original Message----- From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman Sent: Thursday, March 09, 2006 7:28 AM To: SIGMETRICS at LISTSERV.UTK.EDU Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Loet, That is pretty basic, but the problem is that due to Bradford's Law and Garfield's Law definition of such sets is impossible. You are always going to have exogenous variables if you use citations. If you going to use citations in evaluations, they must be used together with other variables--the best being experta ratings if such are available. Then you can check for extreme outliers indicating sources of distortion. The evaluation must be specific to those scientists being evaluated. It is not possible define mathematically sets universally applicable. The one thing that really bothers me about European research is that they seem to assume that citations are valid measures of quality. It then concentrates on find some mathematical technique supposedly capable of measuring quality. This research seems woefully short of studies of the opinions of actual scientists as well as the institutional and social bases of citations. It seems to boil down to fascination with new gimmickry--latest being the present fad with the Hirsch index, Compared with the work done by the American Council on Education and the US National Research Council it is quite crude--even the vaunted British RAE. What the Americans have found is that no matter how carefully you do it, you always crap it up somehow. SB Loet Leydesdorff @LISTSERV.UTK.EDU> on 03/09/2006 12:25:45 AM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 > What's not to laugh about it? There probably is no better way to do > it. > > SB Let me then repeat the problem: comparisons in terms of impact factors, etc., are valid only within cognitive domains with common citation and publication practices. In other words, citation graphs among journals have different densities and this affects the impact factors in the corresponding domains. For example, impact factors of immunology journals are much higher than impact factors of toxicology journals. The delineation of the sets in which one can compare thus matters. We know that this delineation cannot be perfect, but it matters how good they are. Increasingly evaluation commission and scientometric researchers seem to assume that the ISI subject categories are valid delineation of domains within which one can make comparisons. The article by Bollen et al. was a point in case. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ From quentinburrell at MANX.NET Thu Mar 9 15:26:01 2006 From: quentinburrell at MANX.NET (Quentin L. Burrell) Date: Thu, 9 Mar 2006 20:26:01 -0000 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: I have been following these various exchanges with much interest. Let me pick up a subthread from one of Stephen's earlier pieces, quoted in part: ----- Original Message ----- From: "Stephen J Bensman" To: Sent: Monday, March 06, 2006 5:41 PM Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Small as this may be, the > probabilities and lambda were actually much smaller, for Garfield's > constant is based on the set of articles actually cited that year, i.e., > it > it truncated on the left and does not take into account the articles that > could have been cited but were not. I do not have the technical or > intellectual ability to estimate this zero class. I do know that Sir > Maurice Kendall backed off from the problem when he confronted it in > Bradford's Law, and who the hell am I compared to Maurice Kendall. I wish > that somebody would write an article understandable to simpletons on how > to > make such estimates. From my perspective, this would be one of the most > important articles ever written. > The estimation of the zero class is a longstanding problem that I recently referred to in a Letter to the Editor of JASIS&T ("Sample-size dependence or time dependence of statistical measures in informetrics?" 55(2), 183-184, 2004). The relevant extract and some historical references are as follows: "Yoshikane et al. (2003a, 2003b) also make reference to Good (1953), Good & Toulmin (1956), and Efron & Thisted (1976) in the context of interpolation and extrapolation of data. The extrapolation problem - for instance, given the cumulated data for 1992-1997, what can we say about the distribution if the cumulation were to be extended to cover 1998? - has a long history. Within bibliometrics Kendall (1960), in his discussion of Bradford's work on journal productivity, posed the problem "there is also a non-observed class of journals which have not carried a relevant article in the period examined but may do so at any moment in the future. One would like to be able to estimate the size of this potentially contributory class". This problem is equivalent to the so-called unseen species problem in ecology and dates back at least to Fisher et al. (1943), see also Engen (1978). In the ecological context the extrapolation may be in the sense of widening the geographical area, in bibliometrics it is in the sense of increasing the time scale of observation - obviously there are other variants. Kendall's problem was addressed by Brookes (1975) who essentially, but independently, demonstrated a special case of the so-called Good & Toulmin formula (see Burrell (1988)). Efron & Thisted's (1976) empirical approach was further developed and applied within bibliometrics by Burrell (1989, 1990). The current setting of a database being cumulated over time, using the Burrell (1992a) data, was addressed by Burrell (1992b). References Brookes, B. C. (1975). A sampling theorem for finite discrete distributions. Journal of Documentation, 31, 26-35. Burrell, Q. L. (1988). A simple empirical method for predicting library circulations. Journal of Documentation, 44, 302-314. Burrell, Q. L. (1989). On the growth of bibliographies with time: an exercise in bibliometric prediction. Journal of Documentation, 45, 302-317. Burrell, Q. L. (1990). Empirical prediction of library circulations based on negative binomial processes. In L. Egghe & R. Rousseau (Eds.), Informetrics 89/90: Selection of papers submitted for the Second International Conference on Bibliometrics, Scientometrics and Informetrics (pp. 57-64). Amsterdam: Elsevier. Burrell, Q. L. (1992a). The dynamic nature of bibliometric processes: a case study. In I. K. Ravichandra Rao (Ed.), Informetrics - 91: selected papers from the Third International Conference on Informetrics (pp. 97-129), Bangalore: Ranganathan Endowment. Burrell, Q. L. (1992b). One-step-ahead prediction for a growing database: an empirical Bayes approach. Journal of Scientific and Industrial Research, 51, 756-762. Burrell, Q . L. (2003). The sample size dependency of statistical measures in informetrics? Some comments. Journal of the American Society for Information Science and Technology. (Published online 12 June, 2003.) Efron, B. & Thisted, R. (1976). Estimating the number of unseen species: How many words did Shakespeare know? Biometrika, 63, 435-477. Engen, S. (1978). Stochastic abundance models. London: Chapman and Hall. Fisher, R. A., Corbet, A. S. & Williams, C. B. (1943). The relation between the number of species and the number of individuals in a random sample from an animal population. Journal of Animal Ecology, 12, 42-58. Good, I. J. (1953). The population frequencies of species and the estimation of population parameters. Biometrika, 40, 237-264. Good, I. J. & Toulmin, G. H. (1956). The number of new species, and the increase in population coverage, when a sample is increased. Biometrika, 43, 45-63. Kendall, M. G. (1960). The bibliography of operational research. Operational Research Quarterly, 11, 31-36. Yoshikane, F., Kageura, K. & Tsuji, K. (2003a). A method for the comparative analysis of concentration of author productivity, giving consideration to the effect of sample size dependency of statistical measures. Journal of the American Society for Information Science and Technology, 54, 521-528. Yoshikane, F., Kageura, K. & Tsuji, K. (2003b). The sample size dependency of statistical measures and synchronic potentiality in informetrics. Some comments on some comments by Professor Burrell. Journal of the American Society for Information Science and Technology, (Published online 25 June, 2003.)" I am not sure that any of these could be described as "one of the most important article ever written", from whatever perspective, but at least they show that the problem has been considered and some results derived - and applied. Sorry for this diversion from the main theme! Quentin *********************************************** Dr Quentin L Burrell Isle of Man International Business School The Nunnery Old Castletown Road Douglas Isle of Man IM2 1QB via United Kingdom q.burrell at ibs.ac.im www.ibs.ac.im From notsjb at LSU.EDU Thu Mar 9 15:44:55 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Thu, 9 Mar 2006 14:44:55 -0600 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Message-ID: Hooray! I got a rise out of the guy I was going to name as probably the only person active today who could possibly solve this problem. However, instead of giving a history and bibliography of the thing, why don't you write a practitioner piece designed for idiots, explaining in simple mathematicl terms--kindergarten, if possible--on how a practitioner like I can handle the problem. I am tired of always having to dodge around the issue due to my stupidity, and the abililty to estimate the invisible zero class is very important in the practical management of library collections and electronic databases. SB "Quentin L. Burrell" @LISTSERV.UTK.EDU> on 03/09/2006 02:26:01 PM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 I have been following these various exchanges with much interest. Let me pick up a subthread from one of Stephen's earlier pieces, quoted in part: ----- Original Message ----- From: "Stephen J Bensman" To: Sent: Monday, March 06, 2006 5:41 PM Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 Small as this may be, the > probabilities and lambda were actually much smaller, for Garfield's > constant is based on the set of articles actually cited that year, i.e., > it > it truncated on the left and does not take into account the articles that > could have been cited but were not. I do not have the technical or > intellectual ability to estimate this zero claqs. I do know that Sir > Maurice Kendall backed off from the prmblem when he confronted it in > Bradford's Law, and who the hell am I compared to Maurice Kendall. I wish > that somebody would write an article understandable to simpletons on how > to > make such estimates. From my perspective, this would be one of the most > important articles ever written. > The estimation of the zero class is a longstanding problem that I recently referred to in a Letter to the Editor of JASIS&T ("Sample-size dependence or time dependence of statistical measures in informetrics?" 55(2), 183-184, 2004). The relevant extract and some historical references are as follows: "Yoshikane et al. (2003a, 2003b) also make reference to Good (1953), Good & Toulmin (1956), and Efron & Thisted (1976) in the context of interpolation and extrapolation of data. The extrapolation problem - for instance, given the cumulated data for 1992-1997, what can we say about the distribution if the cumulation were to be extended to cover 1998? - has a long history. Within bibliometrics Kendall (1960), in his discussion of Bradford's work on journal productivity, posed the problem "there is also a non-observed class of journals which have not carried a relevant article in the period examined but may do so at any moment in the future. One would like to be able to estimate the size of this potentially contributory class". This problem is equivalent to the so-called unseen species problem in ecology and dates back at least to Fisher et al. (1943), see also Engen (1978). In the ecological context the extrapolation may be in the sense of widening the geographical area, in bibliometrics it is in the sense of increasing the time scale of observation - obviously there are other variants. Kendall's problem was addressed by Brookes (1975) who essentially, but independently, demonstrated a special case of the so-called Good & Toulmin formula (see Burrell (1988)). Efron & Thisted's (1976) empirical approach was further developed and applied within bibliometrics by Burrell (1989, 1990). The current setting of a database being cumulated over time, using the Burrell (1992a) data, was addressed by Burrell (1992b). References Brookes, B. C. (1975). A sampling theorem for finite discrete distributions. Journal of Documentation, 31, 26-35. Burrell, Q. L. (1988). A simple empirical method for predicting library circulations. Journal of Documentation, 44, 302-314. Burrell, Q. L. (1989). On the growth of bibliographies with time: an exercise in bibliometric prediction. Journal of Documentation, 45, 302-317. Burrell, Q. L. (1990). Empirical prediction of library circulations based on negative binomial processes. In L. Egghe & R. Rousseau (Eds.), Informetrics 89/90: Selection of papers submitted for the Second International Conference on Bibliometrics, Scientometrics and Informetrics (pp. 57-64). Amsterdam: Elsevier. Burrell, Q. L. (1992a). The dynamic nature of bibliometric processes: a case study. In I. K. Ravichandra Rao (Ed.), Informetrics - 91: selected papers from the Third International Conference on Informetrics (pp. 97-129), Bangalore: Ranganathan Endowment. Burrell, Q. L. (1992b). One-step-ahead prediction for a growing database: an empirical Bayes approach. Journal of Scientific and Industrial Research, 51, 756-762. Burrell, Q . L. (2003). The sample size dependency of statistical measures in informetrics? Some comments. Journal of the American Society for Information Science and Technology. (Published online 12 June, 2003.) Efron, B. & Thisted, R. (1976). Estimating the number of unseen species: How many words did Shakespeare know? Biometrika, 63, 435-477. Engen, S. (1978). Stochastic abundance models. London: Chapman and Hall. Fisher, R. A., Corbet, A. S. & Williams, C. B. (1943). The relation between the number of species and the number of individuals in a random sample from an animal population. Journal of Animal Ecology, 12, 42-58. Good, I. J. (1953). The population frequencies of species and the estimation of population parameters. Biometrika, 40, 237-264. Good, I. J. & Toulmin, G. H. (1956). The number of new species, and the increase in population coverage, when a sample is increased. Biometrika, 43, 45-63. Kendall, M. G. (1960). The bibliography of operational research. Operational Research Quarterly, 11, 31-36. Yoshikane, F., Kageura, K. & Tsuji, K. (2003a). A method for the comparative analysis of concentration of author productivity, giving consideration to the effect of sample size dependency of statistical measures. Journal of the American Society for Information Science and Technology, 54, 521-528. Yoshikane, F., Kageura, K. & Tsuji, K. (2003b). The sample size dependency of statistical measures and synchronic potentiality in informetrics. Some comments on some comments by Professor Burrell. Journal of the American Society for Information Science and Technology, (Published online 25 June, 2003.)" I am not sure that any of these could be described as "one of the most important article ever written", from whatever perspective, but at least they show that the problem has been considered and some results derived - and applied. Sorry for this diversion from the main theme! Quentin *********************************************** Dr Quentin L Burrell Isle of Man International Business School The Nunnery Old Castletown Road Douglas Isle of Man IM2 1QB via United Kingdom q.burrell at ibs.ac.im www.ibs.ac.im From george at LIBRARY.CALTECH.EDU Thu Mar 9 18:57:37 2006 From: george at LIBRARY.CALTECH.EDU (George Porter) Date: Thu, 9 Mar 2006 15:57:37 -0800 Subject: FW: Dana Roth on citation indexing Message-ID: [Forwarding on Behalf of Dana Roth. -- George] In answer to Sam Brooks' concern about "missing the most complete citation indexes for business, communication & mass media, nursing & allied health, and sociology" ... my article was specifically aimed at 'competitors to the Science Citation Index and the Web of Science.' In answer to Stevan Harnad ... the Citebase website specifically warns that "Citebase is currently only an experimental demonstration. Users are cautioned not to use it for academic evaluation yet. Citation coverage and analysis is incomplete and hit coverage and analysis is both incomplete and noisy." Dana L. Roth Millikan Library / Caltech 1-32 1200 E. California Blvd. Pasadena, CA 91125 626-395-6423 fax 626-792-7540 dzrlib at library.caltech.edu http://library.caltech.edu/collections/chemistry.htm -----Original Message----- From: George Porter [mailto:george at library.caltech.edu] Sent: Wednesday, March 08, 2006 10:53 PM To: DIG_REF; STS-L; ERIL-L; SPARC Open Access Forum; SIGMETRICS at LISTSERV.UTK.EDU Subject: [STS-L] Dana Roth on citation indexing My colleague, Dana Roth, has published an overview of the variety of fee and free citation searching services which have sprung up over the years. Dana L. Roth. (2005) The emergence of competitors to the Science Citation Index and the Web of Science. Current Science 89(9):1531-6 (10 November 2005) His contribution was part of a "Special Section on 50 Years of Citation Indexing". Current Science , just as all of the other publications of the Indian Academy of Sciences , is an Open Access journal. George S. Porter Sherman Fairchild Library of Engineering & Applied Science California Institute of Technology Mail Code 1-43, Pasadena, CA 91125-4300 Telephone (626) 395-3409 Fax (626) 431-2681 http://library.caltech.edu contributor http://stlq.info | http://www.earlham.edu/~peters/fos/fosblog.html ACRL Science & Technology Section Discussion List Join, change your subscription, or unsubscribe: http://lists.ala.org/wws/info/sts-l Archives: http://lists.ala.org/wws/arc/sts-l From loet at LEYDESDORFF.NET Fri Mar 10 06:53:04 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Fri, 10 Mar 2006 12:53:04 +0100 Subject: Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: <200603081631.16234.havemanf@cms.hu-berlin.de> Message-ID: This was my first comment. One should have taken rank-order correlations instead of Pearson correlations. The result based on Table 1 of the article are highly significant :-). I added total citations as a third variable because somewhere in this conversation someone raised the idea that the PageRank would correlate with total citations. This is not the case. With best wishes, Loet _____ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR) Kloveniersburgwal 48, 1012 CX Amsterdam Tel.: +31-20- 525 6598; fax: +31-20- 525 3681 loet at leydesdorff.net ; http://www.leydesdorff.net/ The Knowledge-Based Economy: Modeled, Measured, and Simulated The Self-Organization of the Knowledge-Based Society; The Challenge of Scientometrics -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: clip_image002.gif Type: image/gif Size: 7024 bytes Desc: not available URL: From eugene.garfield at THOMSON.COM Fri Mar 10 10:54:57 2006 From: eugene.garfield at THOMSON.COM (Eugene Garfield) Date: Fri, 10 Mar 2006 10:54:57 -0500 Subject: FW: CHINA SCIENCE AND TECHNOLOGY ASSESSMENT REPORT - 2 Message-ID: FROM: Dr. Ronald N. Kostoff (Office of Naval Research) Kostoff, Ronald [KOSTOFR at ONR.NAVY.MIL] SUBJ: China Science and Technology Assessment Report A report on the structure and infrastructure of Chinese science and technology (1) is now available for downloading (http://www.onr.navy.mil/sci_tech/special/354/technowatch/textmine.asp). Highlights of the report include: OUTPUT PERFORMANCE * China's output of research articles has expanded dramatically in the last decade (articles published in the Science Citation Index) * China is among the research output leaders, especially in critical technologies (e.g., nanotechnology, energetic materials) * China's major research collaborators are, in order: USA, Japan, Germany, England, Canada, Australia * China-USA collaboration emphasizes biomedical first and nanotechnology second, whereas China-Japan collaboration reverses this priority * There are critical research and technology sub-areas where China leads the USA in absolute numbers of research articles published. In these areas, China has at least four times the relative investment emphasis as the USA, since total USA articles are four times the number of total China articles. * Relative to the USA, China emphasizes the hard sciences that underpin defense and commercial needs * Relative to China, the USA emphasizes medical, psychological, and social problem research areas * China's research articles can be assigned to four major categories: Physics/ Materials (13966 records); Life Sciences (7377); Mathematics (7162); Chemistry (5841) CITATION IMPACT PERFORMANCE * Chinese researchers publish in many Chinese journals, but cite very few Chinese journals * Chinese researchers publish in low Impact Factor journals, but cite relatively high Impact Factor journals (Impact Factor measures a journal's ability to attract citations) * Chinese researchers publish in much lower Impact Factor journals than do USA researchers * China-USA collaboration doubles citation impact of Chinese authors * China's research impact was larger than India's in all major research categories (Physical/ Environmental/ Materials/ Life Sciences, as measured by median of top ten cited articles in each sub-area, for technical sub-areas of similar research output) * China's research impact was smaller than Australia's in all major research categories * Global nanotechnology researchers cite only a handful of Chinese journals with significant numbers, and even these journals have two orders of magnitude less citations than the leading international journals As the above and other studies have shown, aggregate country publication productivity and citation impact results can be somewhat misleading. Publications and citation impact in critical technologies and sub-technologies are most important, and should serve as the basis for publication and citation comparison. RNK REFERENCES 1. Kostoff RN, Briggs, MB, Rushenberg, RL, Bowles, CA, Pecht, M. The structure and infrastructure of Chinese science and technology. DTIC Technical Report ADA443315 (http://www.dtic.mil/). Defense Technical Information Center. Fort Belvoir, VA. 2006. A downloadable version of the report's final draft is available at http://www.onr.navy.mil/sci_tech/special/354/technowatch/textmine.asp Go to ninth report listed. Click on PDF version. 2. The views in this report are solely those of the authors, and do not represent the views of the Department of the Navy or any of its components, DDL-OMNI Engineering, LLC, or the University of Maryland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Christina.Pikas at JHUAPL.EDU Fri Mar 10 12:30:54 2006 From: Christina.Pikas at JHUAPL.EDU (Pikas, Christina K.) Date: Fri, 10 Mar 2006 12:30:54 -0500 Subject: FW: CHINA SCIENCE AND TECHNOLOGY ASSESSMENT REPORT - 2 Message-ID: Hi All- I've been reviewing the methodology of the mentioned report and I'm troubled about a couple of key points. First, the citations were retrieved from only EI Compendex and SCI. I believe it's acknowledged that SCI is Western-biased (see for example, a discussion in D. J. de Solla Price's 1986 book). In other words, wouldn't using only these two databases to view the scientific productivity of China really paint an inaccurate picture? Also, I would expect India to be less underrepresented in SCI because scientific communications in India are frequently in English (due to post-colonial factors?) and so there is increased intellectual access to journals written in English. I believe Dr. Leydesdorff and Dr. Zhou used a Chinese-language, Chinese-produced scientific database to examine the representation of journals and citations in SCI. One of their findings was the Chinese journal articles cited western journal articles but western journal articles did not (for the most part) cite Chinese journals. Accordingly, the list of Chinese journals (like Table ES2) provided in the reference is not very meaningful. One other note: it appears that they've used automatic clustering and text matching to assign subject categories. I suppose a similar clustering technique might be used to solve the conundrum of ISI's assigned subject categories? Do others have similar problems or am I not getting it? Thanks, Christina K. Pikas, MLS R.E. Gibson Library & Information Center The Johns Hopkins University Applied Physics Laboratory Voice 240.228.4812 (Washington), 443.778.4812 (Baltimore) Fax 443.778.5353 ________________________________ From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Eugene Garfield Sent: Friday, March 10, 2006 10:55 AM To: SIGMETRICS at LISTSERV.UTK.EDU Subject: [SIGMETRICS] FW: CHINA SCIENCE AND TECHNOLOGY ASSESSMENT REPORT - 2 FROM: Dr. Ronald N. Kostoff (Office of Naval Research) Kostoff, Ronald [KOSTOFR at ONR.NAVY.MIL] SUBJ: China Science and Technology Assessment Report A report on the structure and infrastructure of Chinese science and technology (1) is now available for downloading (http://www.onr.navy.mil/sci_tech/special/354/technowatch/textmine.asp). Highlights of the report include: OUTPUT PERFORMANCE * China's output of research articles has expanded dramatically in the last decade (articles published in the Science Citation Index) * China is among the research output leaders, especially in critical technologies (e.g., nanotechnology, energetic materials) * China's major research collaborators are, in order: USA, Japan, Germany, England, Canada, Australia * China-USA collaboration emphasizes biomedical first and nanotechnology second, whereas China-Japan collaboration reverses this priority * There are critical research and technology sub-areas where China leads the USA in absolute numbers of research articles published. In these areas, China has at least four times the relative investment emphasis as the USA, since total USA articles are four times the number of total China articles. * Relative to the USA, China emphasizes the hard sciences that underpin defense and commercial needs * Relative to China, the USA emphasizes medical, psychological, and social problem research areas * China's research articles can be assigned to four major categories: Physics/ Materials (13966 records); Life Sciences (7377); Mathematics (7162); Chemistry (5841) CITATION IMPACT PERFORMANCE * Chinese researchers publish in many Chinese journals, but cite very few Chinese journals * Chinese researchers publish in low Impact Factor journals, but cite relatively high Impact Factor journals (Impact Factor measures a journal's ability to attract citations) * Chinese researchers publish in much lower Impact Factor journals than do USA researchers * China-USA collaboration doubles citation impact of Chinese authors * China's research impact was larger than India's in all major research categories (Physical/ Environmental/ Materials/ Life Sciences, as measured by median of top ten cited articles in each sub-area, for technical sub-areas of similar research output) * China's research impact was smaller than Australia's in all major research categories * Global nanotechnology researchers cite only a handful of Chinese journals with significant numbers, and even these journals have two orders of magnitude less citations than the leading international journals As the above and other studies have shown, aggregate country publication productivity and citation impact results can be somewhat misleading. Publications and citation impact in critical technologies and sub-technologies are most important, and should serve as the basis for publication and citation comparison. RNK REFERENCES 1. Kostoff RN, Briggs, MB, Rushenberg, RL, Bowles, CA, Pecht, M. The structure and infrastructure of Chinese science and technology. DTIC Technical Report ADA443315 (http://www.dtic.mil/). Defense Technical Information Center. Fort Belvoir, VA. 2006. A downloadable version of the report's final draft is available at http://www.onr.navy.mil/sci_tech/special/354/technowatch/textmine.asp Go to ninth report listed. Click on PDF version. 2. The views in this report are solely those of the authors, and do not represent the views of the Department of the Navy or any of its components, DDL-OMNI Engineering, LLC, or the University of Maryland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From loet at LEYDESDORFF.NET Fri Mar 10 14:57:10 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Fri, 10 Mar 2006 20:57:10 +0100 Subject: FW: CHINA SCIENCE AND TECHNOLOGY ASSESSMENT REPORT - 2 In-Reply-To: <934BB0B6D8A02C42BC6099FDE8149CCD3B48A9@aplesjustice.dom1.jhuapl.edu> Message-ID: Dear Christina, I downloaded the report, but then saw that it has 504 pages. The executive summary alone is 60 pages. Ron Kostoff really is a prolific author and he is also a good friend. For the recent paper in Research Policy 35(1) (2006) 83-104, Ping Zhou and I used also SCI data. We did not find the same strength for China as Ron, but still considerably. We also used 2004 data because it takes a while before a paper is published. I believe Dr. Leydesdorff and Dr. Zhou used a Chinese-language, Chinese-produced scientific database to examine the representation of journals and citations in SCI. One of their findings was the Chinese journal articles cited western journal articles but western journal articles did not (for the most part) cite Chinese journals. Accordingly, the list of Chinese journals (like Table ES2) provided in the reference is not very meaningful. The reference is: Zhou, Ping & Loet Leydesdorff, A Comparison between the China Scientific and Technical Papers and Citations Database and the Science Citation Index in terms of journal hierarchies and inter-journal citation relations. Journal of the American Society for Information Science and Technology (forthcoming); < pdf-version> One other note: it appears that they've used automatic clustering and text matching to assign subject categories. I suppose a similar clustering technique might be used to solve the conundrum of ISI's assigned subject categories? Unfortunately, this is not so easy. For example, there are many clustering algorithms and one has to reason why one should use the one or the other. Obvious, you and I share some interests because I wrote also on this subject about a year ago: Can Scientific Journals be Classified in terms of Aggregated Journal-Journal Citation Relations using the Journal Citation Reports? Journal of the American Society for Information Science and Technology (forthcoming). < pdf-version> With best wishes, Loet Do others have similar problems or am I not getting it? Thanks, Christina K. Pikas, MLS R.E. Gibson Library & Information Center The Johns Hopkins University Applied Physics Laboratory Voice 240.228.4812 (Washington), 443.778.4812 (Baltimore) Fax 443.778.5353 _____ From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Eugene Garfield Sent: Friday, March 10, 2006 10:55 AM To: SIGMETRICS at LISTSERV.UTK.EDU Subject: [SIGMETRICS] FW: CHINA SCIENCE AND TECHNOLOGY ASSESSMENT REPORT - 2 FROM: Dr. Ronald N. Kostoff (Office of Naval Research) Kostoff, Ronald [KOSTOFR at ONR.NAVY.MIL] SUBJ: China Science and Technology Assessment Report A report on the structure and infrastructure of Chinese science and technology (1) is now available for downloading (http://www.onr.navy.mil/sci_tech/special/354/technowatch/textmine.asp). Highlights of the report include: OUTPUT PERFORMANCE * China's output of research articles has expanded dramatically in the last decade (articles published in the Science Citation Index) * China is among the research output leaders, especially in critical technologies (e.g., nanotechnology, energetic materials) * China's major research collaborators are, in order: USA, Japan, Germany, England, Canada, Australia * China-USA collaboration emphasizes biomedical first and nanotechnology second, whereas China-Japan collaboration reverses this priority * There are critical research and technology sub-areas where China leads the USA in absolute numbers of research articles published. In these areas, China has at least four times the relative investment emphasis as the USA, since total USA articles are four times the number of total China articles. * Relative to the USA, China emphasizes the hard sciences that underpin defense and commercial needs * Relative to China, the USA emphasizes medical, psychological, and social problem research areas * China's research articles can be assigned to four major categories: Physics/ Materials (13966 records); Life Sciences (7377); Mathematics (7162); Chemistry (5841) CITATION IMPACT PERFORMANCE * Chinese researchers publish in many Chinese journals, but cite very few Chinese journals * Chinese researchers publish in low Impact Factor journals, but cite relatively high Impact Factor journals (Impact Factor measures a journal's ability to attract citations) * Chinese researchers publish in much lower Impact Factor journals than do USA researchers * China-USA collaboration doubles citation impact of Chinese authors * China's research impact was larger than India's in all major research categories (Physical/ Environmental/ Materials/ Life Sciences, as measured by median of top ten cited articles in each sub-area, for technical sub-areas of similar research output) * China's research impact was smaller than Australia's in all major research categories * Global nanotechnology researchers cite only a handful of Chinese journals with significant numbers, and even these journals have two orders of magnitude less citations than the leading international journals As the above and other studies have shown, aggregate country publication productivity and citation impact results can be somewhat misleading. Publications and citation impact in critical technologies and sub-technologies are most important, and should serve as the basis for publication and citation comparison. RNK REFERENCES 1. Kostoff RN, Briggs, MB, Rushenberg, RL, Bowles, CA, Pecht, M. The structure and infrastructure of Chinese science and technology. DTIC Technical Report ADA443315 (http://www.dtic.mil/). Defense Technical Information Center. Fort Belvoir, VA. 2006. A downloadable version of the report's final draft is available at http://www.onr.navy.mil/sci_tech/special/354/technowatch/textmine.asp Go to ninth report listed. Click on PDF version. 2. The views in this report are solely those of the authors, and do not represent the views of the Department of the Navy or any of its components, DDL-OMNI Engineering, LLC, or the University of Maryland. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dgoodman at PRINCETON.EDU Fri Mar 10 15:18:28 2006 From: dgoodman at PRINCETON.EDU (David Goodman) Date: Fri, 10 Mar 2006 15:18:28 -0500 Subject: comments, Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006 In-Reply-To: Message-ID: An earlier posting is correct, that minor differences in ranking based on IF are statistically meaningless, especially near the bottom. This is related to the choice of non-parametric statistic, and everything I know about that I learned from Bensman's papers; in this context http://www.garfield.library.upenn.edu/bensman/jasistprobart.pdf . When examing a ranked list in a subject, the safest way is quartiles; some publishers I am told use 1/ 3 , 2/3 ; I usually use top 1/10 and top 1/2. This to a certain extent gives a way of comparison between different subjects. One must still bear in mind that in some subjects the really minor journals are included, and in other they are not. In biological subjects at least, I think this correponds to actual human empircal decisions-- the top tenth usually corresopond to the general recognition of the essential journals; the top half is the titles which are generally subscribed to by all major libraries; the bottom half is where only libraries with a special interest would get. And remember there is a substraum of English language Western-country journals which are not even included, and a penubra of non-English language journals or English non-Western countries's journals which ought to be in, and would increase the validity of JCR -based work. A useful device when dealing with journals published irregularly, which can be seen readily enough by the variation in cited article counts is to take a moving average (I find 2-years sufficient.) If one wishes to avoid zeros in one's equation, a convenient practice is to transform all the values by add +1 (article or citation, as needed) If you read the online information in how title changes are handled, it essentially comes down to the inability to use the data for the following two years and the preceeding two years, unless you add the values up yourself. Before you can do this safely you must look at the actual journals; fortunately , this can be done online regardless of access, , because the tables of contents are generally enough. One of the running threads of this discussion seems to have been the practice of using instruments that one does not fully understand. I will say a little about Ulrich's next week. (One could also add the practice of librarians with insufficient statisical knowledge to select and make proper use of the right instrument. But librarians generally have enough sense to know to work together with a statistician when this becomes non-trivial, wheras a few information scientist seem to assume they know everything pertaining to information, with all other professions in a lesser role..) Dr. David Goodman Associate Professor Palmer School of Library and Information Science Long Island University and formerly Princeton University Library dgoodman at liu.edu dgoodman at princeton.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From dgoodman at PRINCETON.EDU Fri Mar 10 15:37:53 2006 From: dgoodman at PRINCETON.EDU (David Goodman) Date: Fri, 10 Mar 2006 15:37:53 -0500 Subject: FW: Dana Roth on citation indexing In-Reply-To: <2DB0071B2579A2448430A96C451BBB27B1D458@clsx.cls.caltech.edu> Message-ID: As for competitors, see http://www.charlestonco.com/comp.cfm?id=43 and now updated by http://www.charlestonco.com/comp.cfm?id=55 There is also Scirus, a competitor to Google Scholar, where I am on the advisory board http://www.scirus.com These products have nothing resembling JCR, which is a pity, for the offer considerable wider geographical coverage. There is reason to doubt that they ever will, (rest of sentence, offline) Dr. David Goodman Associate Professor Palmer School of Library and Information Science Long Island University and formerly Princeton University Library dgoodman at liu.edu dgoodman at princeton.edu ----- Original Message ----- From: George Porter Date: Thursday, March 9, 2006 7:06 pm Subject: [SIGMETRICS] FW: Dana Roth on citation indexing To: SIGMETRICS at LISTSERV.UTK.EDU > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > [Forwarding on Behalf of Dana Roth. -- George] > > In answer to Sam Brooks' concern about "missing the most complete > citation indexes for business, communication & mass media, nursing & > allied health, and sociology" ... my article was specifically > aimed at > 'competitors to the Science Citation Index and the Web of Science.' > > In answer to Stevan Harnad ... the Citebase website specifically warns > that "Citebase is currently only an experimental demonstration. Users > are cautioned not to use it for academic evaluation yet. Citation > coverage and analysis is incomplete and hit coverage and analysis is > both incomplete and noisy." > > Dana L. Roth > Millikan Library / Caltech 1-32 > 1200 E. California Blvd. Pasadena, CA 91125 > 626-395-6423 fax 626-792-7540 > dzrlib at library.caltech.edu > http://library.caltech.edu/collections/chemistry.htm > > > -----Original Message----- > From: George Porter [mailto:george at library.caltech.edu] > Sent: Wednesday, March 08, 2006 10:53 PM > To: DIG_REF; STS-L; ERIL-L; SPARC Open Access Forum; > SIGMETRICS at LISTSERV.UTK.EDU > Subject: [STS-L] Dana Roth on citation indexing > > > My colleague, Dana Roth, has published an overview of the variety > of fee > and free citation searching services which have sprung up over the > years. > > Dana L. Roth. (2005) The emergence of competitors to the Science > Citation Index and the Web of Science. Current Science 89(9):1531- > 6 (10 > November 2005) > > His contribution was part of a "Special Section on 50 Years of > CitationIndexing". Current Science > , just as all > of the other publications of the Indian Academy of Sciences > , is an Open Access journal. > > George S. Porter > Sherman Fairchild Library of Engineering & Applied Science California > Institute of Technology Mail Code 1-43, Pasadena, CA 91125-4300 > Telephone (626) 395-3409 Fax (626) 431-2681 http://library.caltech.edu > contributor http://stlq.info | > http://www.earlham.edu/~peters/fos/fosblog.html > > ACRL Science & Technology Section Discussion List Join, change your > subscription, or unsubscribe: > http://lists.ala.org/wws/info/sts-l > Archives: > http://lists.ala.org/wws/arc/sts-l > From harnad at ECS.SOTON.AC.UK Fri Mar 10 22:08:57 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Sat, 11 Mar 2006 03:08:57 +0000 Subject: Dana Roth on citation indexing Message-ID: On 9-Mar-06, at 10:45 PM, George Porter wrote (for Dana Roth): >In answer to Stevan Harnad ... the Citebase website specifically warns >that "Citebase is currently only an experimental demonstration. Users >are cautioned not to use it for academic evaluation yet. Citation >coverage and analysis is incomplete and hit coverage and analysis is >both incomplete and noisy." And you think that precisely the same thing cannot be made about citeseer and google scholar? That health warning was responsibly placed there to discourage premature use of an experimental tool as an evaluative instrument, but that certainly doesn't mean that the tool is not every bit as worthy of evaluation as an experimental tool as the others that were evaluated (but did not bother to append a health warning)... http://www.citebase.org/help/#impactwarning Citebase has a number of futuristic features that none of the other citation engines have and it would be a pity of Tim Brody's innovations were overlooked because of his prudence about making sure no one rushes to performance reviews before the OA database comes closer to 100% and the new measures are properly validated. Stevan Harnad From garfield at CODEX.CIS.UPENN.EDU Tue Mar 14 14:55:23 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Tue, 14 Mar 2006 14:55:23 -0500 Subject: Now posted with cited references and e-mail address - Bernstam EV, Herskovic JR, Aphinyanaphongs Y, Aliferis CF, Sriram MG, Hersh WR "Using citation data to improve retrieval from MEDLINE " Message-ID: Elmer V. Bernstam : E-mail Addresses: elmer.v.bernstam at uth.tmc.edu Title: Using citation data to improve retrieval from MEDLINE Author(s): Bernstam EV, Herskovic JR, Aphinyanaphongs Y, Aliferis CF, Sriram MG, Hersh WR Source: JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION 13 (1): 96- 105 JAN-FEB 2006 Document Type: Article Language: English Cited References: 34 Times Cited: 0 Abstract: Objective: To determine whether algorithms developed for the World Wide Web can be applied to the biomedical literature in order to identify articles that are important as well as relevant. Design and Measurements: A direct comparison of eight algorithms: simple PubMed queries, clinical queries (sensitive and specific versions), vector cosine comparison, citation count, journal impact factor, PageRank, and machine learning based on polynomial support vector machines. The objective was to prioritize important articles, defined as being included in a pre- existing bibliography of important literature in surgical oncology. Results: Citation-based algorithms were more effective than noncitation- based algorithms at identifying important articles. The most effective strategies were simple citation count and PageRank, which on average identified over six important articles in the first 100 results compared to 0.85 for the best noncitation-based algorithm (p < 0.001). The authors saw similar differences between citation-based and noncitation-based algorithms at 10, 20, 50, 200, 500, and 1,000 results (p < 0.001). Citation lag affects performance of PageRank more than simple citation count. However, in spite of citation lag, citation-based algorithms remain more effective than noncitation-based algorithms. Conclusion: Algorithms that have proved successful on the World Wide Web can be applied to biomedical information retrieval. Citation-based algorithms can help identify important articles within large sets of relevant results. Further studies are needed to determine whether citation- based algorithms can effectively meet actual user information needs. Addresses: Bernstam EV (reprint author), Univ Texas, Hlth Sci Ctr, Sch Hlth Informat Sci, 7000 Fannin St,Suite 600, Houston, TX 77030 USA Univ Texas, Hlth Sci Ctr, Sch Hlth Informat Sci, Houston, TX 77030 USA Vanderbilt Univ, Dept Biomed Informat, Nashville, TN USA Oregon Hlth & Sci Univ, Dept Med Informat & Clin Epidemiol, Portland, OR USA E-mail Addresses: elmer.v.bernstam at uth.tmc.edu Publisher: HANLEY & BELFUS INC, 210 S 13TH ST, PHILADELPHIA, PA 19107 USA IDS Number: 004AI ISSN: 1067-5027 COMPUTATION RELATED : 2003 *MI HLTH SCI LIB A RES COMM REP U MICH : 1992 *NLM NLM RES LISTS BIBL : 2003 *NLM PUBM CLIN QUER TABL *THOMS ISI J CITATION REPORTS : 2003 *THOMS ISI SCI CIT IND EXP 2004 : 2004 APHINYANAPHONGS Y Text categorization models for high-quality article retrieval in internal medicine JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION 12 : 207 2005 APHINYANAPHONGS Y MED SAN FRANC CA : 2004 BACHMANN LM Identifying diagnostic studies in MEDLINE: Reducing the number needed to read JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION 9 : 653 2002 BAEZAYATES R MODERN INFORM RETRIE : 1999 BORODIN A ACM T INTERNET TECHN 5 : 231 2005 BRIN S WWW7 COMPUTER NETWOR 30 : 107 1998 GARFIELD E Journal impact factor: a brief review CANADIAN MEDICAL ASSOCIATION JOURNAL 161 : 979 1999 GARFIELD E CITATION INDEXING AU 1 : 1977 HAYNES RB ACP J CLUB 142 : A8 2005 HAYNES RB DEVELOPING OPTIMAL SEARCH STRATEGIES FOR DETECTING CLINICALLY SOUND STUDIES IN MEDLINE JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION 1 : 447 1994 HERSH WR How well do physicians use electronic information retrieval systems? A framework for investigation and systematic review JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 280 : 1347 1998 HERSH WR MED INFORMATICS COMP : 539 2000 HERSH WR SIGIR 94 DUBL IR : 1994 HERSKOVIC JR AM MED INF ASS FALL : 2005 JOACHIMS T SIGIR WORKSH MATH FO : 2002 KING DN THE CONTRIBUTION OF HOSPITAL LIBRARY INFORMATION-SERVICES TO CLINICAL CARE - A STUDY IN 8 HOSPITALS BULLETIN OF THE MEDICAL LIBRARY ASSOCIATION 75 : 291 1987 KLEINBERG J 9 ANN ACM SIAM S DIS : 1998 LANCASTER F INFORM RETRIEVAL TOD : 1993 MANNING CD FDN STAT NATURAL LAN : 1999 MARSHALL JG IMPACT INFORM PROVID : 1991 OPTHOF T Sense and nonsense about the impact factor CARDIOVASCULAR RESEARCH 33 : 1 1997 PAGE L PAGERANK CITATION RA : 1998 STEGMANN J How to evaluate journal impact factors NATURE 390 : 550 1997 WILKINSON R INFORMATION RETRIEVA : 257 1996 WILLIAMS H ZETTAIR SEARCH ENGIN : 2004 WILSON SR USE CRITICAL INCIDEN : 1989 ZHU M JOINT STAT M BIOPH S : 2003 ZIPSER J MEDLINE PUBMED BEYON From garfield at CODEX.CIS.UPENN.EDU Tue Mar 14 15:04:58 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Tue, 14 Mar 2006 15:04:58 -0500 Subject: Rogers DWO, Hendee WR, Orton CG "Scientific citation indices are useful in evaluating medical physicists for promotion and tenure " MEDICAL PHYSICS 33 (1): 1-3 JAN 2006 Message-ID: E-mail Addresses: David W.O. Rogers : drogers at physics.carleton.ca William R. Hendee : whendee at mcw.edu FULL TEXT AVAILABLE AT : http://scitation.aip.org/dbt/dbt.jsp?KEY=MPHYA6&Volume=33&Issue=1 Title: Scientific citation indices are useful in evaluating medical physicists for promotion and tenure Author(s): Rogers DWO, Hendee WR, Orton CG Source: MEDICAL PHYSICS 33 (1): 1-3 JAN 2006 Document Type: Editorial Material Language: English Cited References: 3 Times Cited: 0 Addresses: Rogers DWO (reprint author), Carleton Univ, Dept Phys, Ottawa, ON K1S 5B6 Canada Carleton Univ, Dept Phys, Ottawa, ON K1S 5B6 Canada Med Coll Wisconsin, Milwaukee, WI 53226 USA E-mail Addresses: drogers at physics.carleton.ca, whendee at mcw.edu Publisher: AMER ASSOC PHYSICISTS MEDICINE AMER INST PHYSICS, STE 1 NO 1, 2 HUNTINGTON QUADRANGLE, MELVILLE, NY 11747-4502 USA IDS Number: 004SI ISSN: 0094-2405 Cited References : PATTERSON MS Medical physics top ten MEDICAL PHYSICS 31 : 682 2004 PATTERSON MS Physics in medicine and biology top ten PHYSICS IN MEDICINE AND BIOLOGY 49 : L1 2004 SMOLIN L PHYS TODAY : 56 2005 From pmd8 at CORNELL.EDU Tue Mar 14 15:25:56 2006 From: pmd8 at CORNELL.EDU (Phil Davis) Date: Tue, 14 Mar 2006 15:25:56 -0500 Subject: Does the arXiv lead to higher citations and reduced publisher downloads In-Reply-To: Message-ID: Dear Sigmetrics, I am relatively new to this list, but in keeping with the spirit of sharing new publications, I am including the following report just posted on the arXiv. --Phil Davis Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles? Authors: Philip M. Davis, Michael J. Fromerth Date: March 14, 2006 http://arxiv.org/abs/cs.DL/0603056 An analysis of 2,765 articles published in four math journals from 1997-2005 indicated that articles deposited in the arXiv received 35% more citations on average than non-deposited articles (an advantage of about 1.1 citations per article), and this difference was most pronounced for highly-cited articles. The most plausible explanation was not the Open Access or Early View postulates, but Self-Selection, which has led to higher quality articles being deposited in the arXiv. Yet in spite of their citation advantage, arXiv-deposited articles received 23% fewer downloads from the publisher's website (about 10 fewer downloads per article) in all but the most recent two years after publication. The data suggest that arXiv and the publisher's website may be fulfilling distinct functional needs of the reader. From pmd8 at CORNELL.EDU Tue Mar 14 16:02:05 2006 From: pmd8 at CORNELL.EDU (Phil Davis) Date: Tue, 14 Mar 2006 16:02:05 -0500 Subject: Does the arXiv lead to higher citations and reduced publisher downloads Message-ID: Dear Sigmetrics, I just found out that the following article won't be available until 8pm tonight, to allow it to pass through the arXiv review process. Sorry for the hasty posting. --Phil Dear Sigmetrics, I am relatively new to this list, but in keeping with the spirit of sharing new publications, I am including the following report just posted on the arXiv. --Phil Davis Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles? Authors: Philip M. Davis, Michael J. Fromerth Date: March 14, 2006 http://arxiv.org/abs/cs.DL/0603056 An analysis of 2,765 articles published in four math journals from 1997-2005 indicated that articles deposited in the arXiv received 35% more citations on average than non-deposited articles (an advantage of about 1.1 citations per article), and this difference was most pronounced for highly-cited articles. The most plausible explanation was not the Open Access or Early View postulates, but Self-Selection, which has led to higher quality articles being deposited in the arXiv. Yet in spite of their citation advantage, arXiv-deposited articles received 23% fewer downloads from the publisher's website (about 10 fewer downloads per article) in all but the most recent two years after publication. The data suggest that arXiv and the publisher's website may be fulfilling distinct functional needs of the reader. From harnad at ECS.SOTON.AC.UK Tue Mar 14 19:34:53 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Wed, 15 Mar 2006 00:34:53 +0000 Subject: Does the arXiv lead to higher citations and reduced publisher downloads? In-Reply-To: <200603142323.k2ENNsBd029022@quickgr.its.yale.edu> Message-ID: On Tue, 14 Mar 2006, Phil Davis wrote: > Liblicense, While our study confirms the same citation advantage > reported by others, it does not attribute Open Access as the > cause of more citations, but to Self-Selection. Open Access > therefore may be a result, not a cause, of authors promoting > higher-quality work. > > Does the arXiv lead to higher citations and reduced publisher downloads for > mathematics articles? > Authors: Philip M. Davis, Michael J. Fromerth > Date: March 14, 2006 > http://arxiv.org/abs/cs.DL/0603056 The full text of Phil Davis's paper is not yet accessible, so I can only respond to the abstract. There are many plausible components of the OA advantage, of which self-selection (Quality Bias: QB) is certainly one -- but not the only one, and unlikely to be the principle one, except under a few special conditions. QB is a temporary phenomenon, obviously, disappearing completely at 100% OA. Same is true for the Competitive Advantage (CA) of (comparable) OA papers over non-OA papers in the same journal issue, as well as the Arxiv Advantage (the advantage of appearing jointly in a central, widely consulted repository). Once 100% OA is reached, QB, CA and AA all vanish. (AA vanishes because of OAI interoperability and central harvesting services.) But there are three other components that remain even at 100% OA: Early Access Advantage (EA): The permanent citation boost from earlier access Quality Advantage (QA): The permanent advantage of quality once the playing field has been levelled and affordability/accessibility no longer biases what is and is not accessible Usage Advantage (UA): Average downloads for OA articles are at least double those of non-OA articles OA Impact Advantage = EA + (AA) + (QB) + QA + (CA) + UA http://eprints.ecs.soton.ac.uk/12085/ > An analysis of 2,765 articles published in four math journals > from 1997-2005 indicated that articles deposited in the arXiv > received 35% more citations on average than non-deposited > articles (an advantage of about 1.1 citations per article), and > this difference was most pronounced for highly-cited articles. > The most plausible explanation was not the Open Access or Early > View postulates, but Self-Selection, which has led to higher > quality articles being deposited in the arXiv. Without seeing the full text one cannot be sure of how this was ascertained, but let us assume that it was by correlation (looking at the author's track record, and their comparable non-OA articles, to show that there is a strong correlation between prior author/article citation rates and probability of later self-archiving). There is no doubt at all that this is a causal factor, and indeed it is the example set by the high-quality authors that helps encourage other authors to self-archive. But the only systematic way to show that QB is the *only* component of the OA advantage, or the biggest one, is to test it at all levels of self-archiving, from 1% to 99%. Obviously a citation advantage that persists even as a larger and larger proportion of the research in the field becomes OA is less and less likely to be due to the fact that the best author/articles are the ones being self-archived. And it also has to be tested for articles at all citation levels (i.e., for comparable low, medium, and high-citation articles). The OA advantage is bigger at the higher citation levels, to be sure, but if it is even present at the lower ones, that already shows that QB is unlikely to be the only factor. As to estimating the relative size of the causal contributions of each of the 6 factors -- this will require a more fine-grained analysis, taking into account not only %OA, citation level, and article age, but also article deposit date. Equating average citation levels for the authors and for the specialty domain will be necessary in the comparisons, and a lot of journals will need to be sampled, in diverse fields, to make sure patterns are not specialty-specific. > Yet in spite of > their citation advantage, arXiv-deposited articles received 23% > fewer downloads from the publisher's website (about 10 fewer > downloads per article) in all but the most recent two years after > publication. The data suggest that arXiv and the publisher's > website may be fulfilling distinct functional needs of the > reader. That sounds like the Arxiv Advantage (AA) expressed in the downloads (UA). Apart from total citation counts and downloads, other interesting variables to look at (and compare for OA effects) include: citation latency, citation longevity and other temporal measures; same for downloads; also authority impact (similar to google's PageRank: citations by higher-cited citers count for more), inbreeding/outbreeding coefficients, co-citations, and semantic correlations. Stevan Harnad Hajjem, C., Harnad, S. and Gingras, Y. (2005) Ten-Year Cross-Disciplinary Comparison of the Growth of Open Access and How it Increases Research Citation Impact. IEEE Data Engineering Bulletin 28(4) pp. 39-47. http://eprints.ecs.soton.ac.uk/11688/ From pmd8 at CORNELL.EDU Tue Mar 14 20:31:43 2006 From: pmd8 at CORNELL.EDU (Philip Meir Davis) Date: Tue, 14 Mar 2006 20:31:43 -0500 Subject: Does the arXiv lead to higher citations and reduced publisher downloads? In-Reply-To: Message-ID: Stevan, The paper is now available. Please see the section where we address the three postulates (Open Access, Early View, and Self-Selection). Of the three, Self-Selection was clearly the strongest explanation. If Open Access is partially at work, it appears only to affect the highly-cited articles. Early-View really could not be supported by the data. --Phil > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > On Tue, 14 Mar 2006, Phil Davis wrote: > >> Liblicense, While our study confirms the same citation advantage >> reported by others, it does not attribute Open Access as the >> cause of more citations, but to Self-Selection. Open Access >> therefore may be a result, not a cause, of authors promoting >> higher-quality work. >> >> Does the arXiv lead to higher citations and reduced publisher downloads >> for >> mathematics articles? >> Authors: Philip M. Davis, Michael J. Fromerth >> Date: March 14, 2006 >> http://arxiv.org/abs/cs.DL/0603056 > > The full text of Phil Davis's paper is not yet accessible, so I can only > respond to the abstract. > > There are many plausible components of the OA advantage, of which > self-selection (Quality Bias: QB) is certainly one -- but not the only > one, and unlikely to be the principle one, except under a few special > conditions. QB is a temporary phenomenon, obviously, disappearing > completely at 100% OA. Same is true for the Competitive Advantage (CA) of > (comparable) OA papers over non-OA papers in the same journal issue, > as well as the Arxiv Advantage (the advantage of appearing jointly > in a central, widely consulted repository). > > Once 100% OA is reached, QB, CA and AA all vanish. (AA vanishes because > of OAI interoperability and central harvesting services.) > > But there are three other components that remain even at 100% OA: > > Early Access Advantage (EA): The permanent citation boost from earlier > access > Quality Advantage (QA): The permanent advantage of quality once the > playing field has been levelled and affordability/accessibility no > longer biases what is and is not accessible > Usage Advantage (UA): Average downloads for OA articles are at least > double those of non-OA articles > > OA Impact Advantage = EA + (AA) + (QB) + QA + (CA) + UA > http://eprints.ecs.soton.ac.uk/12085/ > >> An analysis of 2,765 articles published in four math journals >> from 1997-2005 indicated that articles deposited in the arXiv >> received 35% more citations on average than non-deposited >> articles (an advantage of about 1.1 citations per article), and >> this difference was most pronounced for highly-cited articles. >> The most plausible explanation was not the Open Access or Early >> View postulates, but Self-Selection, which has led to higher >> quality articles being deposited in the arXiv. > > Without seeing the full text one cannot be sure of how this was > ascertained, but let us assume that it was by correlation (looking > at the author's track record, and their comparable non-OA articles, to > show that there is a strong correlation between prior author/article > citation rates and probability of later self-archiving). > > There is no doubt at all that this is a causal factor, and indeed it is > the example set by the high-quality authors that helps encourage other > authors to self-archive. > > But the only systematic way to show that QB is the *only* component of > the OA advantage, or the biggest one, is to test it at all levels of > self-archiving, from 1% to 99%. Obviously a citation advantage that > persists even as a larger and larger proportion of the research in the > field becomes OA is less and less likely to be due to the fact that the > best author/articles are the ones being self-archived. > > And it also has to be tested for articles at all citation levels (i.e., > for comparable low, medium, and high-citation articles). The OA > advantage is bigger at the higher citation levels, to be sure, but if it > is even present at the lower ones, that already shows that QB is > unlikely to be the only factor. > > As to estimating the relative size of the causal contributions of each > of the 6 factors -- this will require a more fine-grained analysis, > taking into account not only %OA, citation level, and article age, but > also article deposit date. Equating average citation levels for the > authors and for the specialty domain will be necessary in the > comparisons, and a lot of journals will need to be sampled, in diverse > fields, to make sure patterns are not specialty-specific. > >> Yet in spite of >> their citation advantage, arXiv-deposited articles received 23% >> fewer downloads from the publisher's website (about 10 fewer >> downloads per article) in all but the most recent two years after >> publication. The data suggest that arXiv and the publisher's >> website may be fulfilling distinct functional needs of the >> reader. > > That sounds like the Arxiv Advantage (AA) expressed in the downloads > (UA). > > Apart from total citation counts and downloads, other interesting > variables to look at (and compare for OA effects) include: citation > latency, citation longevity and other temporal measures; same for > downloads; also authority impact (similar to google's PageRank: > citations by higher-cited citers count for more), inbreeding/outbreeding > coefficients, co-citations, and semantic correlations. > > Stevan Harnad > > Hajjem, C., Harnad, S. and Gingras, Y. (2005) Ten-Year > Cross-Disciplinary Comparison of the Growth of Open Access and How it > Increases Research Citation Impact. IEEE Data Engineering Bulletin 28(4) > pp. 39-47. > http://eprints.ecs.soton.ac.uk/11688/ > From harnad at ECS.SOTON.AC.UK Tue Mar 14 22:03:22 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 14 Mar 2006 22:03:22 -0500 Subject: Does the arXiv lead to higher citations and reduced publisher downloads? In-Reply-To: <1137.132.236.155.208.1142386303.squirrel@webmail.cornell.edu> Message-ID: Phil: I think your results are very interesting, but I don't think they have shown that the OA citation advantage (OAA) is all or mostly a self-selection Quality Bias (QB) correlate, rather than being causal. It is still quite plausible that the OAA is a genuine causal factor, but that it has a bigger effect on the high quality/citation end. That could be a fuzzy threshold effect. And at the low/zero end there could be a lot of articles that are just so weak that they're not going to be cited even if you ram them down people's throats! In other words, what I've called "QA" (Quality Advantage) rather than QB (self-selected Quality Bias) could very well still be the true causal factor: Self-Archiving gives the *better* articles a boost -- not an equal linear boost to all articles! At any rate, the jury is definitely still out on the causal components of OAA. I am still pretty convinced intuitively and logically not only that it's causal, but that it's the biggest of the causal factors, though I'm quite ready to believe the effect is stronger for the better articles. Some of the differences in the reported findings may well also be field differences maths vs astro vs physics vs bio and, perhaps even more importantly, differences arising from differences in overall % OA, by field. (Surely self-selection is a less plausible component of an OAA in a field that is 95% OA than in a field that is 5% OA.) But to sort these out we need much bigger Ns tested across many different fields, with different baseline %OA, and looked at within year and within citation range. For the apparent absence of Mike Kurtz's Early Access effect, this *might* be an astro/math difference, or a 100%OA/30%OA difference. Same for the finding of a much smaller download/citation correlation. Stevan On 14-Mar-06, at 8:31 PM, Philip Meir Davis wrote: > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Stevan, > The paper is now available. Please see the section where we > address the > three postulates (Open Access, Early View, and Self-Selection). Of > the > three, Self-Selection was clearly the strongest explanation. If Open > Access is partially at work, it appears only to affect the highly- > cited > articles. Early-View really could not be supported by the data. > --Phil > > > >> Adminstrative info for SIGMETRICS (for example unsubscribe): >> http://web.utk.edu/~gwhitney/sigmetrics.html >> >> On Tue, 14 Mar 2006, Phil Davis wrote: >> >>> Liblicense, While our study confirms the same citation advantage >>> reported by others, it does not attribute Open Access as the >>> cause of more citations, but to Self-Selection. Open Access >>> therefore may be a result, not a cause, of authors promoting >>> higher-quality work. >>> >>> Does the arXiv lead to higher citations and reduced publisher >>> downloads >>> for >>> mathematics articles? >>> Authors: Philip M. Davis, Michael J. Fromerth >>> Date: March 14, 2006 >>> http://arxiv.org/abs/cs.DL/0603056 >> >> The full text of Phil Davis's paper is not yet accessible, so I >> can only >> respond to the abstract. >> >> There are many plausible components of the OA advantage, of which >> self-selection (Quality Bias: QB) is certainly one -- but not the >> only >> one, and unlikely to be the principle one, except under a few special >> conditions. QB is a temporary phenomenon, obviously, disappearing >> completely at 100% OA. Same is true for the Competitive Advantage >> (CA) of >> (comparable) OA papers over non-OA papers in the same journal issue, >> as well as the Arxiv Advantage (the advantage of appearing jointly >> in a central, widely consulted repository). >> >> Once 100% OA is reached, QB, CA and AA all vanish. (AA vanishes >> because >> of OAI interoperability and central harvesting services.) >> >> But there are three other components that remain even at 100% OA: >> >> Early Access Advantage (EA): The permanent citation boost from >> earlier >> access >> Quality Advantage (QA): The permanent advantage of quality once the >> playing field has been levelled and affordability/ >> accessibility no >> longer biases what is and is not accessible >> Usage Advantage (UA): Average downloads for OA articles are at least >> double those of non-OA articles >> >> OA Impact Advantage = EA + (AA) + (QB) + QA + (CA) + UA >> http://eprints.ecs.soton.ac.uk/12085/ >> >>> An analysis of 2,765 articles published in four math journals >>> from 1997-2005 indicated that articles deposited in the arXiv >>> received 35% more citations on average than non-deposited >>> articles (an advantage of about 1.1 citations per article), and >>> this difference was most pronounced for highly-cited articles. >>> The most plausible explanation was not the Open Access or Early >>> View postulates, but Self-Selection, which has led to higher >>> quality articles being deposited in the arXiv. >> >> Without seeing the full text one cannot be sure of how this was >> ascertained, but let us assume that it was by correlation (looking >> at the author's track record, and their comparable non-OA >> articles, to >> show that there is a strong correlation between prior author/article >> citation rates and probability of later self-archiving). >> >> There is no doubt at all that this is a causal factor, and indeed >> it is >> the example set by the high-quality authors that helps encourage >> other >> authors to self-archive. >> >> But the only systematic way to show that QB is the *only* >> component of >> the OA advantage, or the biggest one, is to test it at all levels of >> self-archiving, from 1% to 99%. Obviously a citation advantage that >> persists even as a larger and larger proportion of the research in >> the >> field becomes OA is less and less likely to be due to the fact >> that the >> best author/articles are the ones being self-archived. >> >> And it also has to be tested for articles at all citation levels >> (i.e., >> for comparable low, medium, and high-citation articles). The OA >> advantage is bigger at the higher citation levels, to be sure, but >> if it >> is even present at the lower ones, that already shows that QB is >> unlikely to be the only factor. >> >> As to estimating the relative size of the causal contributions of >> each >> of the 6 factors -- this will require a more fine-grained analysis, >> taking into account not only %OA, citation level, and article age, >> but >> also article deposit date. Equating average citation levels for the >> authors and for the specialty domain will be necessary in the >> comparisons, and a lot of journals will need to be sampled, in >> diverse >> fields, to make sure patterns are not specialty-specific. >> >>> Yet in spite of >>> their citation advantage, arXiv-deposited articles received 23% >>> fewer downloads from the publisher's website (about 10 fewer >>> downloads per article) in all but the most recent two years after >>> publication. The data suggest that arXiv and the publisher's >>> website may be fulfilling distinct functional needs of the >>> reader. >> >> That sounds like the Arxiv Advantage (AA) expressed in the downloads >> (UA). >> >> Apart from total citation counts and downloads, other interesting >> variables to look at (and compare for OA effects) include: citation >> latency, citation longevity and other temporal measures; same for >> downloads; also authority impact (similar to google's PageRank: >> citations by higher-cited citers count for more), inbreeding/ >> outbreeding >> coefficients, co-citations, and semantic correlations. >> >> Stevan Harnad >> >> Hajjem, C., Harnad, S. and Gingras, Y. (2005) Ten-Year >> Cross-Disciplinary Comparison of the Growth of Open Access and How it >> Increases Research Citation Impact. IEEE Data Engineering Bulletin >> 28(4) >> pp. 39-47. >> http://eprints.ecs.soton.ac.uk/11688/ >> From loet at LEYDESDORFF.NET Wed Mar 15 01:34:01 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Wed, 15 Mar 2006 07:34:01 +0100 Subject: Impact Factor and Immediacy Index Message-ID: Dear colleagues, In reaction to my posting last week with the figures in which the Impact Factor, the c/p ratio, and the Immediacy Index were grouped together as a single component, someone noted a previous publication by Weiping Yue, Concepsion S. Wilson, and Ronald Rousseau, "The Immediacy Index and the Journal Impact Factor: Two Highly Correlated Derived Measures," The Canadian Journal of Information and Library Science, 28(1) (2004) 33-48. The authors argue that the II can be used as a predictor for the IF in the next year because they are correlated. With best wishes, Loet _____ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR) Kloveniersburgwal 48, 1012 CX Amsterdam Tel.: +31-20- 525 6598; fax: +31-20- 525 3681 loet at leydesdorff.net ; http://www.leydesdorff.net/ The Knowledge-Based Economy: Modeled, Measured, and Simulated The Self-Organization of the Knowledge-Based Society; The Challenge of Scientometrics -------------- next part -------------- An HTML attachment was scrubbed... URL: From isidro at CINDOC.CSIC.ES Wed Mar 15 07:02:13 2006 From: isidro at CINDOC.CSIC.ES (Isidro F. Aguillo) Date: Wed, 15 Mar 2006 13:02:13 +0100 Subject: 11th International Conference of the International Society for Scientometrics and Informetrics Message-ID: Dear colleague, This is the first Announcement of the 11th International Conference of the International Society for Scientometrics and Informetrics (ISSI) to be held in June 25-27 2007 in Madrid (Spain) hosted by the Spanish Research Council (CSIC). Please, find the First Announcement at the conference website: http://issi2007.cindoc.csic.es Best regards, Isidro F. Aguillo on behalf of the Organising Committee isidro at cindoc.csic.es CINDOC-CSIC Spanish Research Council Joaquin Costa, 22 28002 Madrid. SPAIN http://www.cindoc.csic.es From pmd8 at CORNELL.EDU Wed Mar 15 08:56:12 2006 From: pmd8 at CORNELL.EDU (Phil Davis) Date: Wed, 15 Mar 2006 08:56:12 -0500 Subject: Does the arXiv lead to higher citations and reduced publisher downloads? In-Reply-To: <556513B0-125F-41C3-B976-BD2A3A50532F@ecs.soton.ac.uk> Message-ID: Stevan, Thanks for your thoughtful response. In the absence of a controlled experiment, the best one can do is 1) confirm covariance; 2) confirm temporal order (the cause precedes the effect); 3) confirm a theoretical basis for the phenomena; and 4) systematically rule out all other explanatory causes. I've been able to do all of these except 2) since our data is cumulative in nature. Michael Kurtz however was able to work with temporal data in his study of astrophysics journals, and was unable to confirm the Open Access Postulate. I don't disagree at all that there may be multiple causes working simultaneously, and demonstrating interaction effects. I also don't disagree that the causes may vary for different fields of study. I am however, troubled by individuals who make universal and unqualified statements like, "Open Access increases citations by 50-250%!" The more precise answer is much more subtle, but I understand that a statement like, "open access may provide some citation benefit, but only for prestigious authors who publish in prestigious journals and whose article is already highly-cited", doesn't sound as convincing to administrators and policy makers. M. J. KURTZ, G. EICHHORN, A. ACCOMAZZI, C. GRANT, M. DEMLEITNER, E. HENNEKEN, S. S. MURRAY, The effect of use and access on citations, Information Processing and Management, 41 (2005) 1395-1402. Available: http://arxiv.org/abs/cs.DL/0503029 --Phil >Phil: > >I think your results are very interesting, but I don't think they >have shown that the OA citation advantage (OAA) is all or mostly a >self-selection Quality Bias (QB) correlate, rather than being causal. >It is still quite plausible that the OAA is a genuine causal factor, >but that it has a bigger effect on the high quality/citation end. >That could be a fuzzy threshold effect. And at the low/zero end there >could be a lot of articles that are just so weak that they're not >going to be cited even if you ram them down people's throats! In >other words, what I've called "QA" (Quality Advantage) rather than QB >(self-selected Quality Bias) could very well still be the true causal >factor: Self-Archiving gives the *better* articles a boost -- not an >equal linear boost to all articles! > >At any rate, the jury is definitely still out on the causal >components of OAA. I am still pretty convinced intuitively and >logically not only that it's causal, but that it's the biggest of the >causal factors, though I'm quite ready to believe the effect is >stronger for the better articles. > >Some of the differences in the reported findings may well also be >field differences maths vs astro vs physics vs bio and, perhaps even >more importantly, differences arising from differences in overall % >OA, by field. (Surely self-selection is a less plausible component of >an OAA in a field that is 95% OA than in a field that is 5% OA.) But >to sort these out we need much bigger Ns tested across many different >fields, with different baseline %OA, and looked at within year and >within citation range. > >For the apparent absence of Mike Kurtz's Early Access effect, this >*might* be an astro/math difference, or a 100%OA/30%OA difference. >Same for the finding of a much smaller download/citation correlation. > >Stevan > >On 14-Mar-06, at 8:31 PM, Philip Meir Davis wrote: > >>Adminstrative info for SIGMETRICS (for example unsubscribe): >>http://web.utk.edu/~gwhitney/sigmetrics.html >> >>Stevan, >>The paper is now available. Please see the section where we >>address the >>three postulates (Open Access, Early View, and Self-Selection). Of >>the >>three, Self-Selection was clearly the strongest explanation. If Open >>Access is partially at work, it appears only to affect the highly- >>cited >>articles. Early-View really could not be supported by the data. >>--Phil >> >> >> >>>Adminstrative info for SIGMETRICS (for example unsubscribe): >>>http://web.utk.edu/~gwhitney/sigmetrics.html >>> >>>On Tue, 14 Mar 2006, Phil Davis wrote: >>> >>>>Liblicense, While our study confirms the same citation advantage >>>>reported by others, it does not attribute Open Access as the >>>>cause of more citations, but to Self-Selection. Open Access >>>>therefore may be a result, not a cause, of authors promoting >>>>higher-quality work. >>>> >>>>Does the arXiv lead to higher citations and reduced publisher >>>>downloads >>>>for >>>>mathematics articles? >>>>Authors: Philip M. Davis, Michael J. Fromerth >>>>Date: March 14, 2006 >>>>http://arxiv.org/abs/cs.DL/0603056 >>> >>>The full text of Phil Davis's paper is not yet accessible, so I >>>can only >>>respond to the abstract. >>> >>>There are many plausible components of the OA advantage, of which >>>self-selection (Quality Bias: QB) is certainly one -- but not the >>>only >>>one, and unlikely to be the principle one, except under a few special >>>conditions. QB is a temporary phenomenon, obviously, disappearing >>>completely at 100% OA. Same is true for the Competitive Advantage >>>(CA) of >>>(comparable) OA papers over non-OA papers in the same journal issue, >>>as well as the Arxiv Advantage (the advantage of appearing jointly >>>in a central, widely consulted repository). >>> >>>Once 100% OA is reached, QB, CA and AA all vanish. (AA vanishes >>>because >>>of OAI interoperability and central harvesting services.) >>> >>>But there are three other components that remain even at 100% OA: >>> >>>Early Access Advantage (EA): The permanent citation boost from >>>earlier >>>access >>>Quality Advantage (QA): The permanent advantage of quality once the >>> playing field has been levelled and affordability/ >>>accessibility no >>> longer biases what is and is not accessible >>>Usage Advantage (UA): Average downloads for OA articles are at least >>> double those of non-OA articles >>> >>> OA Impact Advantage = EA + (AA) + (QB) + QA + (CA) + UA >>> http://eprints.ecs.soton.ac.uk/12085/ >>> >>>>An analysis of 2,765 articles published in four math journals >>>>from 1997-2005 indicated that articles deposited in the arXiv >>>>received 35% more citations on average than non-deposited >>>>articles (an advantage of about 1.1 citations per article), and >>>>this difference was most pronounced for highly-cited articles. >>>>The most plausible explanation was not the Open Access or Early >>>>View postulates, but Self-Selection, which has led to higher >>>>quality articles being deposited in the arXiv. >>> >>>Without seeing the full text one cannot be sure of how this was >>>ascertained, but let us assume that it was by correlation (looking >>>at the author's track record, and their comparable non-OA >>>articles, to >>>show that there is a strong correlation between prior author/article >>>citation rates and probability of later self-archiving). >>> >>>There is no doubt at all that this is a causal factor, and indeed >>>it is >>>the example set by the high-quality authors that helps encourage >>>other >>>authors to self-archive. >>> >>>But the only systematic way to show that QB is the *only* >>>component of >>>the OA advantage, or the biggest one, is to test it at all levels of >>>self-archiving, from 1% to 99%. Obviously a citation advantage that >>>persists even as a larger and larger proportion of the research in >>>the >>>field becomes OA is less and less likely to be due to the fact >>>that the >>>best author/articles are the ones being self-archived. >>> >>>And it also has to be tested for articles at all citation levels >>>(i.e., >>>for comparable low, medium, and high-citation articles). The OA >>>advantage is bigger at the higher citation levels, to be sure, but >>>if it >>>is even present at the lower ones, that already shows that QB is >>>unlikely to be the only factor. >>> >>>As to estimating the relative size of the causal contributions of >>>each >>>of the 6 factors -- this will require a more fine-grained analysis, >>>taking into account not only %OA, citation level, and article age, >>>but >>>also article deposit date. Equating average citation levels for the >>>authors and for the specialty domain will be necessary in the >>>comparisons, and a lot of journals will need to be sampled, in >>>diverse >>>fields, to make sure patterns are not specialty-specific. >>> >>>>Yet in spite of >>>>their citation advantage, arXiv-deposited articles received 23% >>>>fewer downloads from the publisher's website (about 10 fewer >>>>downloads per article) in all but the most recent two years after >>>>publication. The data suggest that arXiv and the publisher's >>>>website may be fulfilling distinct functional needs of the >>>>reader. >>> >>>That sounds like the Arxiv Advantage (AA) expressed in the downloads >>>(UA). >>> >>>Apart from total citation counts and downloads, other interesting >>>variables to look at (and compare for OA effects) include: citation >>>latency, citation longevity and other temporal measures; same for >>>downloads; also authority impact (similar to google's PageRank: >>>citations by higher-cited citers count for more), inbreeding/ >>>outbreeding >>>coefficients, co-citations, and semantic correlations. >>> >>>Stevan Harnad >>> >>>Hajjem, C., Harnad, S. and Gingras, Y. (2005) Ten-Year >>>Cross-Disciplinary Comparison of the Growth of Open Access and How it >>>Increases Research Citation Impact. IEEE Data Engineering Bulletin >>>28(4) >>>pp. 39-47. >>>http://eprints.ecs.soton.ac.uk/11688/ From harnad at ECS.SOTON.AC.UK Wed Mar 15 11:12:54 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Wed, 15 Mar 2006 16:12:54 +0000 Subject: Does the arXiv lead to higher citations and reduced publisher downloads? In-Reply-To: <6.2.1.2.2.20060315084532.02288178@postoffice8.mail.cornell.edu> Message-ID: On Wed, 15 Mar 2006, Phil Davis wrote: > Thanks for your thoughtful response. In the absence of a controlled > experiment, the best one can do is 1) confirm covariance; Agreed. > 2) confirm temporal order (the cause precedes the effect); Not yet methodologically feasible, alas. > 3) confirm a theoretical basis for the phenomena; There are several plausible theories for an OA Advantage (OAA), compatible with the evidence to date. They don't all agree, but, as I said, there are likely to be multiple causal factors. (Future research will have to measure both their causality and their relative size.) > and 4) systematically rule out all other explanatory causes. This too certainly has not yet been done by anyone. > I've been able to do all of these except 2) since our data is cumulative > in nature. Not just not (2), but also not (4), especially in light of the fact that more than one explanation is compatible with your findings (3)... > Michael Kurtz however was able to work with > temporal data in his study of astrophysics journals, and was unable to > confirm the Open Access Postulate. I know about and admire his work. But astro is anomalous in that it is 100% OA and became that way at one fell swoop (via ADS). No intermediate temporal stages in which 25% then 50% then 75% of astro became OA, hence no way to test the Competitive Advantage (CA). Mike cites three causal factors: QB (author self-selection Quality Bias) and EA (strong Early Access effects, which you failed to find in your maths data) and UA (Usage Advantage: doubled downloads -- you did not report comparative totals, but only compared Arxiv to publisher downloads). Mike found no residual increased citations overall owing to 100% OA (in fact, somewhat fewer!) and -- I think correctly -- interpreted this as being because when *everything* is OA -- 100% OA, a level playing field -- then authors do not cite *more*: they cite more *selectively*, based on importance and relevance, rather than unconsciously biassed, as in most other fields, by affordability/accessibility constraints. That might shift the citations around (from a less relevant, accessible article that I might have cited before I had access to all articles to a more relevant article that I can now access) but it does not increase the total number of citations per article I write (in fact, it decreases it slightly). (This is one manifestation of what I called QA, the Quality Advantage, though a very hard one to pin down causally from the kinds of data available: It almost requires an author study on citation strategies before and after. I would say your high-end correlation between downloads and citations is indirect evidence in support of it.) What the astro data miss are the effects of CA, the Competitive Advantage of OA vs. non-OA articles within the same journal and issue, and similar comparisons of like with like (e.g., same citation band) because astro has the boon and bain, as noted, of having gone 100% OA in one swoop. It hence also lacks a means of testing the other form of QA, in which the better articles get selectively advantaged if they are OA compared to when they are not OA. (Correlations, citation bands, and other ways of equating comparable content will need to be used to test this in fields that are not yet 100% OA, hence allowing a basis for comparison!) I am still betting most of my money on CA, QA, and EA (as well as UA) rather than just the QB (and AA) that your article stresses. Also in the download/citation correlation. > I don't disagree at all that there may be multiple causes working > simultaneously, and demonstrating interaction effects. I also don't > disagree that the causes may vary for different fields of study. I am > however, troubled by individuals who make universal and unqualified > statements like, "Open Access increases citations by 50-250%!" I am such an individual, and I hold by that statement (though certainly not because I mean to trouble you!) > The more > precise answer is much more subtle, but I understand that a statement like, > "open access may provide some citation benefit, but only for prestigious > authors who publish in prestigious journals and whose article is already > highly-cited", doesn't sound as convincing to administrators and policy > makers. Not just that, but I don't think it is a correct statement of the thrust of the current body of findings on the OAA. It is merely your interpretation of the result of your own study in 4 maths journals! Stevan > M. J. KURTZ, G. EICHHORN, A. ACCOMAZZI, C. GRANT, M. DEMLEITNER, E. > HENNEKEN, S. S. MURRAY, The effect of use and access on citations, > Information Processing and Management, 41 (2005) 1395-1402. Available: > http://arxiv.org/abs/cs.DL/0503029 > > >I think your results are very interesting, but I don't think they > >have shown that the OA citation advantage (OAA) is all or mostly a > >self-selection Quality Bias (QB) correlate, rather than being causal. > >It is still quite plausible that the OAA is a genuine causal factor, > >but that it has a bigger effect on the high quality/citation end. > >That could be a fuzzy threshold effect. And at the low/zero end there > >could be a lot of articles that are just so weak that they're not > >going to be cited even if you ram them down people's throats! In > >other words, what I've called "QA" (Quality Advantage) rather than QB > >(self-selected Quality Bias) could very well still be the true causal > >factor: Self-Archiving gives the *better* articles a boost -- not an > >equal linear boost to all articles! > > > >At any rate, the jury is definitely still out on the causal > >components of OAA. I am still pretty convinced intuitively and > >logically not only that it's causal, but that it's the biggest of the > >causal factors, though I'm quite ready to believe the effect is > >stronger for the better articles. > > > >Some of the differences in the reported findings may well also be > >field differences maths vs astro vs physics vs bio and, perhaps even > >more importantly, differences arising from differences in overall % > >OA, by field. (Surely self-selection is a less plausible component of > >an OAA in a field that is 95% OA than in a field that is 5% OA.) But > >to sort these out we need much bigger Ns tested across many different > >fields, with different baseline %OA, and looked at within year and > >within citation range. > > > >For the apparent absence of Mike Kurtz's Early Access effect, this > >*might* be an astro/math difference, or a 100%OA/30%OA difference. > >Same for the finding of a much smaller download/citation correlation. > > > >Stevan > > > >On 14-Mar-06, at 8:31 PM, Philip Meir Davis wrote: > > > >>The paper is now available. Please see the section where we > >>address the > >>three postulates (Open Access, Early View, and Self-Selection). Of > >>the > >>three, Self-Selection was clearly the strongest explanation. If Open > >>Access is partially at work, it appears only to affect the highly- > >>cited > >>articles. Early-View really could not be supported by the data. > >>--Phil > >>> > >>>On Tue, 14 Mar 2006, Phil Davis wrote: > >>> > >>>>Liblicense, While our study confirms the same citation advantage > >>>>reported by others, it does not attribute Open Access as the > >>>>cause of more citations, but to Self-Selection. Open Access > >>>>therefore may be a result, not a cause, of authors promoting > >>>>higher-quality work. > >>>> > >>>>Does the arXiv lead to higher citations and reduced publisher > >>>>downloads > >>>>for > >>>>mathematics articles? > >>>>Authors: Philip M. Davis, Michael J. Fromerth > >>>>Date: March 14, 2006 > >>>>http://arxiv.org/abs/cs.DL/0603056 > >>> > >>>The full text of Phil Davis's paper is not yet accessible, so I > >>>can only > >>>respond to the abstract. > >>> > >>>There are many plausible components of the OA advantage, of which > >>>self-selection (Quality Bias: QB) is certainly one -- but not the > >>>only > >>>one, and unlikely to be the principle one, except under a few special > >>>conditions. QB is a temporary phenomenon, obviously, disappearing > >>>completely at 100% OA. Same is true for the Competitive Advantage > >>>(CA) of > >>>(comparable) OA papers over non-OA papers in the same journal issue, > >>>as well as the Arxiv Advantage (the advantage of appearing jointly > >>>in a central, widely consulted repository). > >>> > >>>Once 100% OA is reached, QB, CA and AA all vanish. (AA vanishes > >>>because > >>>of OAI interoperability and central harvesting services.) > >>> > >>>But there are three other components that remain even at 100% OA: > >>> > >>>Early Access Advantage (EA): The permanent citation boost from > >>>earlier > >>>access > >>>Quality Advantage (QA): The permanent advantage of quality once the > >>> playing field has been levelled and affordability/ > >>>accessibility no > >>> longer biases what is and is not accessible > >>>Usage Advantage (UA): Average downloads for OA articles are at least > >>> double those of non-OA articles > >>> > >>> OA Impact Advantage = EA + (AA) + (QB) + QA + (CA) + UA > >>> http://eprints.ecs.soton.ac.uk/12085/ > >>> > >>>>An analysis of 2,765 articles published in four math journals > >>>>from 1997-2005 indicated that articles deposited in the arXiv > >>>>received 35% more citations on average than non-deposited > >>>>articles (an advantage of about 1.1 citations per article), and > >>>>this difference was most pronounced for highly-cited articles. > >>>>The most plausible explanation was not the Open Access or Early > >>>>View postulates, but Self-Selection, which has led to higher > >>>>quality articles being deposited in the arXiv. > >>> > >>>Without seeing the full text one cannot be sure of how this was > >>>ascertained, but let us assume that it was by correlation (looking > >>>at the author's track record, and their comparable non-OA > >>>articles, to > >>>show that there is a strong correlation between prior author/article > >>>citation rates and probability of later self-archiving). > >>> > >>>There is no doubt at all that this is a causal factor, and indeed > >>>it is > >>>the example set by the high-quality authors that helps encourage > >>>other > >>>authors to self-archive. > >>> > >>>But the only systematic way to show that QB is the *only* > >>>component of > >>>the OA advantage, or the biggest one, is to test it at all levels of > >>>self-archiving, from 1% to 99%. Obviously a citation advantage that > >>>persists even as a larger and larger proportion of the research in > >>>the > >>>field becomes OA is less and less likely to be due to the fact > >>>that the > >>>best author/articles are the ones being self-archived. > >>> > >>>And it also has to be tested for articles at all citation levels > >>>(i.e., > >>>for comparable low, medium, and high-citation articles). The OA > >>>advantage is bigger at the higher citation levels, to be sure, but > >>>if it > >>>is even present at the lower ones, that already shows that QB is > >>>unlikely to be the only factor. > >>> > >>>As to estimating the relative size of the causal contributions of > >>>each > >>>of the 6 factors -- this will require a more fine-grained analysis, > >>>taking into account not only %OA, citation level, and article age, > >>>but > >>>also article deposit date. Equating average citation levels for the > >>>authors and for the specialty domain will be necessary in the > >>>comparisons, and a lot of journals will need to be sampled, in > >>>diverse > >>>fields, to make sure patterns are not specialty-specific. > >>> > >>>>Yet in spite of > >>>>their citation advantage, arXiv-deposited articles received 23% > >>>>fewer downloads from the publisher's website (about 10 fewer > >>>>downloads per article) in all but the most recent two years after > >>>>publication. The data suggest that arXiv and the publisher's > >>>>website may be fulfilling distinct functional needs of the > >>>>reader. > >>> > >>>That sounds like the Arxiv Advantage (AA) expressed in the downloads > >>>(UA). > >>> > >>>Apart from total citation counts and downloads, other interesting > >>>variables to look at (and compare for OA effects) include: citation > >>>latency, citation longevity and other temporal measures; same for > >>>downloads; also authority impact (similar to google's PageRank: > >>>citations by higher-cited citers count for more), inbreeding/ > >>>outbreeding > >>>coefficients, co-citations, and semantic correlations. > >>> > >>>Stevan Harnad > >>> > >>>Hajjem, C., Harnad, S. and Gingras, Y. (2005) Ten-Year > >>>Cross-Disciplinary Comparison of the Growth of Open Access and How it > >>>Increases Research Citation Impact. IEEE Data Engineering Bulletin > >>>28(4) > >>>pp. 39-47. > >>>http://eprints.ecs.soton.ac.uk/11688/ > From garfield at CODEX.CIS.UPENN.EDU Wed Mar 15 17:14:03 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Wed, 15 Mar 2006 17:14:03 -0500 Subject: Bensman SJ Part 2 - "Urquhart's law: Probability and the management of scientific and technical journal collections - Part 2." Science & Technology Libraries, 26 (2). 2005. p.5-31. Haworth Press Inc. Binghamton Message-ID: FULL TEXT AVAILABLE AT : http://garfield.library.upenn.edu/bensman/urquhartlaw2.pdf also available parts 1 and 3 http://garfield.library.upenn.edu/bensman/urquhartlaw1.pdf http://garfield.library.upenn.edu/bensman/urquhartlaw3.pdf Full text posted with the permission of The Haworth Press (http://www.haworthpress.com/) - publisher of scholarly and professional books and journals.Copyright 2005 by The Haworth Press, Inc. All rights reserved. E-mail : S.J. Bensman : notsjb at lsu.edu TITLE: Urquhart's law: Probability and the management of scientific and technical journal collections - Part 2.Probability in the development and management of a central document delivery collection AUTHOR: Bensman, SJ SOURCE: SCIENCE & TECHNOLOGY LIBRARIES 26 (2). 2005. p.5-31 HAWORTH PRESS INC, BINGHAMTON ABSTRACT: Part I of this three-part paper discussed the genesis of the law formulated by Donald J. Urquhart on the use of scientific and technical (sci/tech) journals through interlibrary loan and central document delivery. It concentrated on the study of such use conducted by Urquhart in preparation for the establishment of the National Lending Library for Science and Technology (NLL), which became the central document delivery library of Britain. The focus of Part I was on the statistical and probabilistic bases of this law. In Part 2 the emphasis shifts to how Urquhart utilized probability to develop and manage the NLL's sci/tech journal collection. Urquhart based his collection development and management policies on a high-loan core of journals, causing the stability of sci/tech journal use over time to become the main theoretical issue. Part 2 analyzes the controversy over this issue between Urquhart and his successor, Maurice B. Line. AUTHOR ADDRESS: SJ Bensman, Louisiana State Univ, LSU Lib, Baton Rouge, LA 70803 USA From loet at LEYDESDORFF.NET Thu Mar 16 16:41:23 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Thu, 16 Mar 2006 22:41:23 +0100 Subject: Journal-journal citations for Chinese journals (ISTIC) in 2004 now online Message-ID: ?????????????? ????????????????????????/?????Pajek?????? ?????? ? ?????? ???????????? ? ?????????/??????????? 2003 2004 ???? ???? ???? ???? One can click on any of the journal names in the corresponding box and obtain the Pajek file corresponding to the citation environment of the journal ("citing" or "cited"). See for further explanation: Zhou Ping & Loet Leydesdorff, ?????????????? [Visualization of the Citation Impact Environments in the CSTPC Journal Set] with a manual at < ???? ??????/??????????> , ???????? , Chinese Journal of Scientific and Technical Periodicals, 16(6) (2005) 773-780. 2003 2004 cited journal files cited journal files citing journal files citing journal files _____ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR) Kloveniersburgwal 48, 1012 CX Amsterdam Tel.: +31-20- 525 6598; fax: +31-20- 525 3681 loet at leydesdorff.net ; http://www.leydesdorff.net/ The Knowledge-Based Economy: Modeled, Measured, and Simulated The Self-Organization of the Knowledge-Based Society; The Challenge of Scientometrics -------------- next part -------------- An HTML attachment was scrubbed... URL: From garfield at CODEX.CIS.UPENN.EDU Mon Mar 20 14:34:24 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Mon, 20 Mar 2006 14:34:24 -0500 Subject: Bensman SJ. Part 3 - Urquhart's law: Probability and the management of scientific and technical journal collections - The law's final formulation and implications for library systems" Science & Technology Libraries 26(2). p.33-69, 2005. Message-ID: FULL TEXT AVAILABLE AT : http://garfield.library.upenn.edu/bensman/urquhartlaw3.pdf also available parts 1 and 2 http://garfield.library.upenn.edu/bensman/urquhartlaw1.pdf http://garfield.library.upenn.edu/bensman/urquhartlaw2.pdf Full text posted with the permission of The Haworth Press (http://www.haworthpress.com/) - publisher of scholarly and professional books and journals.Copyright 2005 by The Haworth Press, Inc. All rights reserved. E-mail : S.J. Bensman : notsjb at lsu.edu TITLE: Urquhart's law: Probability and the management of scientific and technical journal collections - Part 3. The law's final formulation and implications for library systems AUTHOR: Bensman, SJ SOURCE: SCIENCE & TECHNOLOGY LIBRARIES 26 (2). 2005. p.33-69 HAWORTH PRESS INC, BINGHAMTON ABSTRACT: This is the last part of an analysis of the law formulated by Donald J. Urquhart in respect to the use of scientific and technical (sci/tech) journals through either interlibrary loan or central document delivery. The first part discussed the genesis of this law as well as its statistical bases; the second dealt with the implications of the law for the development and management of the sci/tech journal collection of a central document delivery library. Part 3 concentrates on its consequences for all the libraries of a given library system. There is analyzed in this part the controversy over the validity of this law that Caused Urquhart to formulate it in an explicit manner. Part 3 discusses Studies at various libraries that have corroborated this law, postulating that this law forces a probabilistic re-conceptualization of the functioning, of the sci/tech journal system. Throughout this part, there are noted the consequences of the law for such matters as sci/tech journal sales, resource sharing, document delivery, interlibrary loan, and the transition from individually held paper copies of journals to shared electronic databases. AUTHOR ADDRESS: SJ Bensman, Louisiana State Univ, LSU Lib, Baton Rouge, LA 70803 USA From garfield at CODEX.CIS.UPENN.EDU Mon Mar 20 16:29:49 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Mon, 20 Mar 2006 16:29:49 -0500 Subject: Abt HA. "National Astronomical Productivities" Bulletin of the American Astronomical Society Volume 37 Issue 4, p.1540-1543, 2005. Message-ID: Helmut A. Abt : abt at noao.edu AUTHOR : H.A. Abt TITLE : National Astronomical Productivities SOURCE : Bulletin of the American Astronomical Society Volume 37 Issue 4, p.1540-1543, 2005. EXCERPT: INTRODUCTION: Astronomical research is no longer limited to a few European and North American countries, but originates in at least 55 countries. How do those countries compare in their support and productivity of original astronomical research? Of course we would not expect developing countries to contribute much because they have more basic needs. Education in astronomy is important in all countries in our understanding of the world around us, but original research, which is expensive, should generally be done in developing countries only after more urgent human needs are satisfied. However, there are 30 countries whose per capita incomes are greater than $4,000 (US) that should be able to afford to contribute to our knowledge of astronomy. In this study the numbers of papers published by astronomers from various countries in 13 of the world's foremost astronomical journals are compared with the Gross National Products (GNP) of those countries. The values of the GNPs are usually available from almanacs (e.g. Wright 2004). The countries of origin for the publications are more tedious to determine. Typically a paper has five authors from three different countries. Of the various addresses often given for some authors, we usually selected the country where the research was done. Then it is simple arithmetic to conclude, for instance in this example, that two of the countries each receive credit for 0.40 of a paper and one for 0.20 of a paper. Cases of 40 authors from 12 different countries involve more arithmetic, and credits can be as low as 0.01 or 0.02 of a paper. From garfield at CODEX.CIS.UPENN.EDU Mon Mar 20 16:53:11 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Mon, 20 Mar 2006 16:53:11 -0500 Subject: A.G. Kelly - Monographs on Informetrics, Information Research, 10(4), review #R181, 2005. Review of Leo Egghe's "Power laws in the information production process: Lotkaian informetrics" . Message-ID: FULL REVIEW AVAILABLE AT : http://informationr.net/ir/reviews/revs181.html AUTHOR : Kelly AG TITLE : Monographs on informetrics SOURCE : Information Research, 10(4), review #R181, 2005. The Author has reviewed the following : Egghe, Leo "Power laws in the information production process: Lotkaian informetrics" Kidlington, Oxfordshire: Elsevier Academic Press, 2005. xvii, 427 p. ISBN 0- 12-088753-3. ?68.99. This monograph is devoted to explaining the regularities encountered in literature (information production) using Lotka's size frequency function. Lotka's law is treated as a tool that can be used for various informetric issues helping to explain certain regularities in the light of earlier results without always starting from zero (p. 295). From garfield at CODEX.CIS.UPENN.EDU Mon Mar 20 16:58:52 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Mon, 20 Mar 2006 16:58:52 -0500 Subject: A.G. Kelly - Monographs on Informetrics, Information Research, 10(4), review #R181, 2005. Review of Mike Thelwall's "Link analysis: an information science approach" San Diego, CA: Elsevier Academic Press, 2004. Message-ID: FULL REVIEW AVAILABLE AT : http://informationr.net/ir/reviews/revs181.html AUTHOR : Kelly AG TITLE : Monographs on informetrics SOURCE : Information Research, 10(4), review #R181, 2005. The Author has reviewed the following : Thelwall, Mike. Link analysis: an information science approach. San Diego, CA: Elsevier Academic Press, 2004. xxx p. ISBN 0-12-088553-0. ?43.99. Thelwall's book is of a different character and form. First: the book is hybrid as it is partly online and partly printed. It includes text, software and data. Apart from this hardcover version, the price of which is mentioned above, there is also a freely accessible part on the Web. The online users may not even be aware that the book exists, but the readers of the book will be guided to the online part, which is constantly updated. From loet at LEYDESDORFF.NET Thu Mar 23 04:31:56 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Thu, 23 Mar 2006 10:31:56 +0100 Subject: extensions of FullText.exe and Ti.ex for co-word analysis with provisions for stop word lists and word frequency lists Message-ID: Dear colleagues, While using the programs in class, it became clear that one needs a provision to generate word frequency lists from the texts and to correct for stop words. The latter facility is now added to these programs (at http://www.leydesdorff.net/software/ti/stopword.exe ) and the former is set as a hyperlink to TextSTAT-2 of Dutch Linguistics Department of the Technical University in Berlin (at http://www.niederlandistik.fu-berlin.de/textstat/software-en.html). The programs themselves can be found at http://www.leydesdorff.net/software/ti and http://www.leydesdorff.net/software/fulltext , respectively. For advanced users it may be useful to remark that one can also replace the cosine matrices with Pearson correlation matrices by feeding the output file matrix.dbf into SPSS and running the appropriate routines. In my opinion, some convincing arguments have been made to use the cosine as the similarity criterion for the visualizations. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR) Kloveniersburgwal 48, 1012 CX Amsterdam Tel.: +31-20- 525 6598; fax: +31-20- 525 3681 loet at leydesdorff.net ; http://www.leydesdorff.net/ From harnad at ECS.SOTON.AC.UK Thu Mar 23 09:47:59 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Thu, 23 Mar 2006 14:47:59 +0000 Subject: Future UK RAEs to be Metrics-Based Message-ID: ** Apologies for Cross-Posting ** Don't say we didn't tell you so! The wasteful, time-consuming RAE will be replaced by metrics, chief among them citation impact, which already correlated with and predicted the RAE outcome anyway, without being explicitly counted. Now it can be explicitly counted (along with other powerful new metrics) and all the rest of the ritualistic time-wasting can be abandoned, without ceremony. This is a great boost for institutional self-archiving in OA Institutional Repositories, not only as the obvious, optimal means of submission, but as the means of maximising research impact: http://irra.eprints.org/software/bronze/ (I hope RCUK is listening!): "Research exercise to be scrapped" Donald MacLeod, Guardian Wednesday March 22, 2006 http://education.guardian.co.uk/RAE/story/0,,1737082,00.html Cf: Harnad, S. (2001) Why I think that research access, impact and assessment are linked. Times Higher Education Supplement 1487: p. 16. http://www.thes.co.uk/search/story.aspx?story_id=74889 http://cogprints.org/1683/ Harnad, S. (2003) Why I believe that all UK research output should be online. Times Higher Education Supplement. Friday, June 6 2003. http://www.thes.co.uk/search/story.aspx?story_id=92599 http://eprints.ecs.soton.ac.uk/7728/ Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives: Improving the UK Research Assessment Exercise whilst making it cheaper and easier. Ariadne. http://www.ariadne.ac.uk/issue35/harnad/ From harnad at ECS.SOTON.AC.UK Thu Mar 23 10:16:44 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Thu, 23 Mar 2006 15:16:44 +0000 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: <05625F047220CF43AED7CA6CBA8B6A7D0F8E19@HERMES1.ds.leeds.ac.uk> Message-ID: On Thu, 23 Mar 2006, Adrian Smith wrote: > See also > http://www.nature.com/nature/journal/v440/n7083/pdf/440408d.pdf > > Mark Maslin [UCL] in "Nature" > Research skewed by stress on highest-impact journals This will "unskew" it, by instead placing the weight on the individual author/article citation counts (and download counts, CiteRanks, authority counts, citation/download latency, citation/longevity, co-citation signature, and many, many new OA metrics waiting to be devised and validated, including full-text semantic-analysis and semantic-web-tag analyses too) rather than just, or primarily, the blunter instrument (the journal impact factor). This is not just about one number any more! The journal tag will still have some weight, but just one weight among many, in an OA scientometric multiple regression equation, customised for each discipline. This is an occasion for rejoicing at progress, pluralism and openness, not digging up obsolescent concerns about over-reliance on the journal impact factor. Stevan Harnad > :"-----Original Message----- > :"From: jisc development discussion forum > :"[mailto:JISC-DEVELOPMENT at JISCMAIL.AC.UK] On Behalf Of Stevan Harnad > :"Sent: 23 March 2006 14:48 > :"To: JISC-DEVELOPMENT at JISCMAIL.AC.UK > :"Subject: Future UK RAEs to be Metrics-Based > :" > :" > :"Don't say we didn't tell you so! The wasteful, time-consuming > :"RAE will be replaced by metrics, chief among them citation > :"impact, which already correlated with and predicted the RAE > :"outcome anyway, without being explicitly counted. Now it can > :"be explicitly counted (along with other powerful new metrics) > :"and all the rest of the ritualistic time-wasting can be > :"abandoned, without ceremony. This is a great boost for > :"institutional self-archiving in OA Institutional > :"Repositories, not only as the obvious, optimal means of > :"submission, but as the means of maximising research impact: > :"http://irra.eprints.org/software/bronze/ > :"(I hope RCUK is listening!): > :" > :" "Research exercise to be scrapped" > :" Donald MacLeod, Guardian Wednesday March 22, 2006 > :" http://education.guardian.co.uk/RAE/story/0,,1737082,00.html > :" > :"Cf: Harnad, S. (2001) Why I think that research access, > :"impact and assessment > :" are linked. Times Higher Education Supplement 1487: p. 16. > :" http://www.thes.co.uk/search/story.aspx?story_id=74889 > :" http://cogprints.org/1683/ > :" > :" Harnad, S. (2003) Why I believe that all UK research output > :" should be online. > :" Times Higher Education Supplement. Friday, June 6 2003. > :" http://www.thes.co.uk/search/story.aspx?story_id=92599 > :" http://eprints.ecs.soton.ac.uk/7728/ > :" > :" Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated > :" online RAE CVs Linked to University Eprint Archives: Improving > :" the UK Research Assessment Exercise whilst making it cheaper and > :" easier. Ariadne. > :" http://www.ariadne.ac.uk/issue35/harnad/ > :" > From harnad at ECS.SOTON.AC.UK Thu Mar 23 15:30:28 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Thu, 23 Mar 2006 20:30:28 +0000 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: <1F2CD7CD-60D9-4B74-B596-0121101167E5@comp.lancs.ac.uk> Message-ID: On Thu, 23 Mar 2006, Ian Sommerville wrote: > This is the wording from the budget document > > "The Government > wants this to continue, but thinks the close correlation between > Research Council income > and QR income may provide an opportunity for allocating QR using a > radically simpler > system. " > > The point is made that, at an institutional level, there is a 0.98 > correlation between research income and QR. No mention of citation > impact. An alternative metric may be proposed for the humanities. You are quite right that the default metric many have in mind is research income, but be patient! Now that the door has been opened to objective metrics, this will spawn more and more candidates for enriching the metric equation. If RAE top-slicing seeks to be an independent funding source, it will want to have some predictive metrics that are independent of prior funding; if it just wants to redundantly follow research funding, it need merely scale up research grants to absorb what would have been the RAE top-slice! The important thing is to scrap the useless, time-wasting RAE preparation/evaluation ritual we were all faithfully performing, when the outcome was already so predictable from other, cheaper sources. Objective metrics are the natural, sensible way to conduct such an exercise, and once we are doing metrics, many powerful new predictive measures will present themselves, over and above grant income and citations. The ranking will not come from one variable, but from a multiple regression equation, with many weighted predictor metrics. Stevan Harnad Professor of Cognitive Science Department of Electronics and Computer Science University of Southampton Highfield, Southampton SO17 1BJ UNITED KINGDOM phone: +44 23-80 592-388 fax: +44 23-80 592-865 harnad at ecs.soton.ac.uk http://www.ecs.soton.ac.uk/~harnad/ > Ian > > > > > On 23 Mar 2006, at 14:47, Stevan Harnad wrote: > > > ** Apologies for Cross-Posting ** > > > > Don't say we didn't tell you so! The wasteful, time-consuming RAE will > > be replaced by metrics, chief among them citation impact, which > > already > > correlated with and predicted the RAE outcome anyway, without being > > explicitly counted. Now it can be explicitly counted (along with other > > powerful new metrics) and all the rest of the ritualistic time-wasting > > can be abandoned, without ceremony. This is a great boost for > > institutional self-archiving in OA Institutional Repositories, not > > only > > as the obvious, optimal means of submission, but as the means of > > maximising research impact: http://irra.eprints.org/software/bronze/ > > (I hope RCUK is listening!): > > > > "Research exercise to be scrapped" > > Donald MacLeod, Guardian Wednesday March 22, 2006 > > http://education.guardian.co.uk/RAE/story/0,,1737082,00.html > > > > Cf: Harnad, S. (2001) Why I think that research access, impact and > > assessment > > are linked. Times Higher Education Supplement 1487: p. 16. > > http://www.thes.co.uk/search/story.aspx?story_id=74889 > > http://cogprints.org/1683/ > > > > Harnad, S. (2003) Why I believe that all UK research output > > should be online. > > Times Higher Education Supplement. Friday, June 6 2003. > > http://www.thes.co.uk/search/story.aspx?story_id=92599 > > http://eprints.ecs.soton.ac.uk/7728/ > > > > Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated > > online RAE CVs Linked to University Eprint Archives: Improving > > the UK Research Assessment Exercise whilst making it cheaper and > > easier. Ariadne. > > http://www.ariadne.ac.uk/issue35/harnad/ > > From dbedford at WORLDBANK.ORG Thu Mar 23 18:39:28 2006 From: dbedford at WORLDBANK.ORG (D. Bedford) Date: Thu, 23 Mar 2006 18:39:28 -0500 Subject: Denise Ann Dowding Bedford is out of the office. Message-ID: I will be out of the office starting 03/23/2006 and will not return until 04/04/2006. I will respond to your message when I return. From harnad at ECS.SOTON.AC.UK Thu Mar 23 20:26:39 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Fri, 24 Mar 2006 01:26:39 +0000 Subject: Does the arXiv lead to higher citations and reduced publisher downloads? In-Reply-To: <200603240047.k2O0lBQ3013073@quickgr.its.yale.edu> Message-ID: On Thu, 23 Mar 2006, Phil Davis wrote: > 1) You call the citation differential an "Open Access Advantage", > thus affirming that access is both the cause and effect of the > phenomena. This is a circular argument. But more importantly, > > 2) Your theory is unrefutable. There's nothing circular here (though there's plenty that's getting repetitious)! If Phil doesn't like to call the OAA the Open Access Advantage, call it the Open Access Numerator Advantage (OANA): Either way, it means, operationally, that when you divide the average citation counts for self-archived articles by the average citation counts for non-self-archived articles in the same journal/issue, the numerator is always bigger. That's the OANA. The rest is about the *cause* of that OANA. Phil likes QB: an author bias toward self-archiving higher-quality, higher-citation articles. I also like QA: increased usage and citation of freely available articles, with greater OANA the higher the quality of the article. No circularity for either inference, but no direct test of causality either, for either inference. Testing causality requires a more direct control and analysis of the timing. For example, if we have deposit dates as well as publication dates, we can look at articles that are self-archived, say, two years after their publication date, and we can equate articles for their citation counts up to the date they were self-archived. If Phil is right, there should be no difference at year three for the citation counts of self-archived articles, compared to control non-self-archived articles that had the same citation counts at year two. If I am right, there should still be a OANA at year three. That is just one of many ways one could test causation. So it's not about irrefutable hypotheses, it's about unreflective interpretations. Stevan Harnad PS Since Arxiv authors tend to deposit preprints before publication and postprints at publication, but rarely postprints much later than publication, we cannot do this particular study in Arxiv; it could be done in CogPrints, but the N's would be small. Once IRs with date-stamps grow, we will be able to do it there. On Thu, 23 Mar 2006, Phil Davis wrote: > Stevan Harnad wrote: > > "What your data show is that the OA Advantage (which everyone > confirms) is stronger on the high-end, and this could either be > because people tend to self-archive high-end articles more (QB), > or because the OA Advantage is stronger on the high end (QA). > Either way it's a quality effect. One way it's a Quality Bias > (QB), the other way it's a Quality Advantage (QA). You think > it's mostly QB, I think it's mostly QA. The data are compatible > with both. More fine-tuned causal tests are needed to decide." > > Phil Responds: > > Stevan, you seem to be referring repeatedly to your web blog, > "Open Access Archivangelism", that > > OA Impact Advantage = EA + (AA) + (QB) + QA + (CA) + UA > http://openaccess.eprints.org/index.php?/archives/29-guid.html > > While I certainly agree that a citation advantage can have > multiple and conflating causes, you call the result of all of > these causes the "Open Access Advantage". In your equation, ALL > possible causes (early view, arXiv, quality bias, quality > advantage, competitive advantage, and usage advantage) can > support your OAA theory. This is problematic on a fundamental > scientific level for two reasons: > > 1) You call the citation differential an "Open Access Advantage", > thus affirming that access is both the cause and effect of the > phenomena. This is a circular argument. But more importantly, > > 2) Your theory is unrefutable. When everything confirms the Open > Access Advantage, OAA becomes meaningless as a scientific theory. > The philosopher of science Karl Popper would have called OAA > "unfalsifiable" and therefore unscientific. > > So if you insist on describing a possible quality differential > between articles deposited in the arXiv and those that are not an > "Open Access Advantage", then I have merely reaffirmed that our > results support OAA. But so does the weather, the stock market, > and my horoscope for today (which by the way, says that I should > avoid Sagittarius) > > --Phil Davis > > From JWS at DB.DK Fri Mar 24 02:52:05 2006 From: JWS at DB.DK (Jesper Wiborg Schneider) Date: Fri, 24 Mar 2006 08:52:05 +0100 Subject: SV: [SIGMETRICS] extensions of FullText.exe and Ti.ex for co-word analysis with provisions for stop word lists and word frequency lists Message-ID: Dear Loet; Many thanks for the programme. I've tried out your program, but I run into difficulties. I'm unable to create .dat files for use with pajek. I use Olle Persson's bibexcel program to extract words from titles. Subsequently, I make a frequency analysis in order to choose the words for the word.txt file in your program - one word pr. Line. Next I arrange all the paper titles in a text file, one title pr. Line and save it with CR/LF as a text.txt file for use with your program. I now run the program. I do get dbase IV output files, but I do not get the .dat files for use with pajek - I wonder where I go wrong? Could you provide an example of a word.txt and a text.txt file - perhaps I can detect my errors? Best wishes - Jesper Schneider ********************************************** Jesper Wiborg Schneider, PhD, Assistant Professor Department of Information Studies Royal School of Library & Information Science Sohng?rdsholmsvej 2, DK-9000 Aalborg, DENMARK Tel. +45 98773041, Fax. +45 98151042 E-mail: jws at db.dk Homepage:http://www2.db.dk/jws/home_dk.htm ********************************************** -----Oprindelig meddelelse----- Fra: Loet Leydesdorff [mailto:loet at LEYDESDORFF.NET] Sendt: 23. marts 2006 10:32 Til: SIGMETRICS at LISTSERV.UTK.EDU Emne: [SIGMETRICS] extensions of FullText.exe and Ti.ex for co-word analysis with provisions for stop word lists and word frequency lists Dear colleagues, While using the programs in class, it became clear that one needs a provision to generate word frequency lists from the texts and to correct for stop words. The latter facility is now added to these programs (at http://www.leydesdorff.net/software/ti/stopword.exe ) and the former is set as a hyperlink to TextSTAT-2 of Dutch Linguistics Department of the Technical University in Berlin (at http://www.niederlandistik.fu-berlin.de/textstat/software-en.html). The programs themselves can be found at http://www.leydesdorff.net/software/ti and http://www.leydesdorff.net/software/fulltext , respectively. For advanced users it may be useful to remark that one can also replace the cosine matrices with Pearson correlation matrices by feeding the output file matrix.dbf into SPSS and running the appropriate routines. In my opinion, some convincing arguments have been made to use the cosine as the similarity criterion for the visualizations. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR) Kloveniersburgwal 48, 1012 CX Amsterdam Tel.: +31-20- 525 6598; fax: +31-20- 525 3681 loet at leydesdorff.net ; http://www.leydesdorff.net/ From loet at LEYDESDORFF.NET Fri Mar 24 03:30:13 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Fri, 24 Mar 2006 09:30:13 +0100 Subject: SV: [SIGMETRICS] extensions of FullText.exe and Ti.ex for co-word analysis with provisions for stop word lists and word frequency lists In-Reply-To: Message-ID: Dear Jesper: Usually, one gets either an error message or the output. However, let us take this offline and not both the list with technical details. I'll send you files shortly. Thanks for trying. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Jesper > Wiborg Schneider > Sent: Friday, March 24, 2006 8:52 AM > To: SIGMETRICS at LISTSERV.UTK.EDU > Subject: [SIGMETRICS] SV: [SIGMETRICS] extensions of > FullText.exe and Ti.ex for co-word analysis with provisions > for stop word lists and word frequency lists > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Dear Loet; > > Many thanks for the programme. > I've tried out your program, but I run into difficulties. I'm > unable to create .dat files for use with pajek. I use Olle > Persson's bibexcel program to extract words from titles. > Subsequently, I make a frequency analysis in order to choose > the words for the word.txt file in your program - one word > pr. Line. Next I arrange all the paper titles in a text file, > one title pr. > Line and save it with CR/LF as a text.txt file for use with > your program. I now run the program. I do get dbase IV output > files, but I do not get the .dat files for use with pajek - I > wonder where I go wrong? > > Could you provide an example of a word.txt and a text.txt > file - perhaps I can detect my errors? > > Best wishes - Jesper Schneider > > ********************************************** > Jesper Wiborg Schneider, PhD, Assistant Professor Department > of Information Studies Royal School of Library & Information > Science Sohng?rdsholmsvej 2, DK-9000 Aalborg, DENMARK Tel. > +45 98773041, Fax. +45 > 98151042 > E-mail: jws at db.dk > Homepage:http://www2.db.dk/jws/home_dk.htm > ********************************************** > > -----Oprindelig meddelelse----- > Fra: Loet Leydesdorff [mailto:loet at LEYDESDORFF.NET] > Sendt: 23. marts 2006 10:32 > Til: SIGMETRICS at LISTSERV.UTK.EDU > Emne: [SIGMETRICS] extensions of FullText.exe and Ti.ex for > co-word analysis with provisions for stop word lists and word > frequency lists > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Dear colleagues, > > While using the programs in class, it became clear that one > needs a provision to generate word frequency lists from the > texts and to correct for stop words. The latter facility is > now added to these programs (at > http://www.leydesdorff.net/software/ti/stopword.exe ) and the > former is set as a hyperlink to TextSTAT-2 of Dutch > Linguistics Department of the Technical University in Berlin > (at > http://www.niederlandistik.fu-berlin.de/textstat/software-en.html). > > The programs themselves can be found at > http://www.leydesdorff.net/software/ti and > http://www.leydesdorff.net/software/fulltext , respectively. > For advanced users it may be useful to remark that one can > also replace the cosine matrices with Pearson correlation > matrices by feeding the output file matrix.dbf into SPSS and > running the appropriate routines. In my opinion, some > convincing arguments have been made to use the cosine as the > similarity criterion for the visualizations. > > With best wishes, > > > Loet > ________________________________ > > Loet Leydesdorff > Amsterdam School of Communications Research (ASCoR) > Kloveniersburgwal 48, 1012 CX Amsterdam > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681 > loet at leydesdorff.net ; http://www.leydesdorff.net/ > From garfield at CODEX.CIS.UPENN.EDU Fri Mar 24 11:54:53 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Fri, 24 Mar 2006 11:54:53 -0500 Subject: Diamond A. "Measurement, incentives and constraints in Stigler=?ISO-8859-1?Q?=92s?= economics of science" Euro.J. History of Economic Thought 12:4 p.635-661, December 2005. Message-ID: Email: adiamond at mail.unomaha.edu AUTHOR : Arthur Diamond TITLE : Measurement, incentives and constraints in Stigler?s economics of science* SOURCE : Euro. J. History of Economic Thought 12:4 635 ? 661 December 2005 ADDRESS : University of Nebraska at Omaha, Nebraska Abstract George J. Stigler?s seminal role as one of the founders of the economics of science is summarized and evaluated. His main contribution rests in his asking an array of important questions and arguing persuasively for the application of empirical, and especially statistical, techniques to the answering of those questions. He asks whether and how science progresses; whether a scientist?s biography is important in understanding his science; what characteristics of a scientist are most complementary to success in science; and how the professionalization of science redirects the attention of scientists more toward internal puzzle-solving, and less toward applied relevance. Includes discussion of Stigler?s contributions to citation analysis. http://cba.unomaha.edu/faculty/adiamond/web/DiamondPDFs/StiglerEJHET.pdf Readers of Sigmetrics are encouraged to visit the web site of Prof. Arthur M. Diamond, a pioneer in he use of citation analysis in the field of economics. The url for his recent paper on the Novelist George Stigler is provided above, but access to many more of his works can be found at his personal web page:: http://cba.unomaha.edu/faculty/adiamond/web/diahompg.htm From garfield at CODEX.CIS.UPENN.EDU Fri Mar 24 12:13:11 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Fri, 24 Mar 2006 12:13:11 -0500 Subject: Correction - Diamond A. "Measurement, incentives and constraints in Stigler=?ISO-8859-1?Q?=92s_economics_of_science_"_Euro._J._History_of_Economic_Thought_12:4_635_=96?= 661 December 2005 Message-ID: In the earlier posting of the article about George J. Stigler, the 1982 winner of the Nobel Prize in Economics, a typographical error identified him as a novelist rather than Nobelist ? an unfortunate slip of the finger. Email: adiamond at mail.unomaha.edu AUTHOR : Arthur Diamond TITLE : Measurement, incentives and constraints in Stigler?s economics of science SOURCE : Euro. J. History of Economic Thought 12:4 635 ? 661 December 2005 ADDRESS : University of Nebraska at Omaha, Nebraska Abstract George J. Stigler?s seminal role as one of the founders of the economics of science is summarized and evaluated. His main contribution rests in his asking an array of important questions and arguing persuasively for the application of empirical, and especially statistical, techniques to the answering of those questions. He asks whether and how science progresses; whether a scientist?s biography is important in understanding his science; what characteristics of a scientist are most complementary to success in science; and how the professionalization of science redirects the attention of scientists more toward internal puzzle-solving, and less toward applied relevance. Includes discussion of Stigler?s contributions to citation analysis. http://cba.unomaha.edu/faculty/adiamond/web/DiamondPDFs/StiglerEJHET.pdf Readers of Sigmetrics are encouraged to visit the web site of Prof. Arthur M. Diamond, a pioneer in he use of citation analysis in the field of economics. The url for his recent paper on the Nobelist George Stigler is provided above, but access to many more of his works can be found at his personal web page:: http://cba.unomaha.edu/faculty/adiamond/web/diahompg.htm From garfield at CODEX.CIS.UPENN.EDU Fri Mar 24 14:30:18 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Fri, 24 Mar 2006 14:30:18 -0500 Subject: Nikolaos A Patsopoulos, Apostolos A Analatos, John P A Ioannidis "Origin and funding of the most frequently cited papers in medicine: database analysis" BMJ, doi:10.1136/bmj.38768.420139.80 (published 17 March 2006) Message-ID: This article is cited by its DOI (digital object identifier) Online First contains the final versions of articles that have been accepted for publication but have not yet appeared in the paper journal. They have been peer reviewed and edited to the journal's usual standards. Some time in the future they will be published in the paper journal, although most likely in abridged form. Online First articles should be cited by their digital object identifiers (DOIs) and date of publication, which are given at the top of the article. Learn more about Online First at : http://bmj.bmjjournals.com/misc/onlinefirst.shtml FULL TEXT OF THE FOLLOWING ARTICLE IS AVAILABLE AT : http://press.psprings.co.uk/bmj/march/patn301531.pdf TITLE : Origin and funding of the most frequently cited papers in medicine: database analysis AUTHOR : Nikolaos A Patsopoulos, Apostolos A Analatos, John P A Ioannidis SOURCE : British Medical Journal (BMJ), doi:10.1136/bmj.38768.420139.80 (published 17 March 2006) Abstract Objective : To evaluate changes in the role of academics and the sources of funding for the medical research cited most frequently over the past decade. Design : Database analysis. Data sources : Web of Knowledge database. Methods : For each year from 1994 to 2003, articles in the domain of clinical medicine that had been cited most often by the end of 2004 were identified. Changes in authors? affiliations and funding sources were evaluated. Results Of the 289 frequently cited articles, most had at least one author with a university (76%) or hospital (57%) affiliation, and the proportion of articles with each type of affiliation was constant over time. Government or public funding was most common (60% of articles), followed by industry (36%). The proportion of most frequently cited articles funded by industry increased over time (odds ratio 1.17 per year, P = 0.001) and was equal to the proportion funded by government or public sources by 2001. 65 of the 77 most cited randomised controlled trials received funding from industry, and the proportion increased significantly over time (odds ratio 1.59 per year, P = 0.003). 18 of the 32 most cited trials published after 1999 were funded by industry alone. Conclusion Academic affiliations remain prominent among the authors of the most frequently cited medical research. Such research is increasingly funded by industry, often exclusively so. Academics may be losing control of the clinical research agenda. Our analysis was one of the evidence tasks (systematic reviews and surveys addressing problems of academic medicine) undertaken as part of the International Campaign to Revitalise Academic Medicine (ICRAM).3 We analysed the affiliations of authors and the funding sources of articles in the domain of clinical medicine that had received the highest number of citations according to the essential science indicators module of the Web of Knowledge database. Our main aim was to see whether the impact of academic institutions and industry has changed during the past decade. Citations do not reflect fully the quality of a paper, but they are a measure of the impact of research.10 Papers that are cited frequently have a major impact on scientific debate and the evolution of scientific thinking, even if citations are not always positive and may even be critical. Methods Identification of the most frequently cited papers We downloaded the most frequently cited papers in clinical medicine of each year since 1994 from the essential science indicators module of the Web of Knowledge produced by the Institute for Scientific Information (ISI). Articles were ranked according to how many times they were cited by any journal indexed by ISI between 1 January 1994 and 31 December 2004. Clinical medicine is one of 21 scientific domains catalogued by ISI and covers all medical sciences and subspecialties. ISI categorises articles across the 21 domains according to the journal of publication; for articles published in multidisciplinary journals, each article is categorised in the domain that most of its citations are derived from. From garfield at CODEX.CIS.UPENN.EDU Fri Mar 24 15:22:30 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Fri, 24 Mar 2006 15:22:30 -0500 Subject: Macias-Chapula CA "Toward a model of communications in public health in Latin America and the Caribbean " REVISTA PANAMERICANA DE SALUD PUBLICA-PAN AMERICAN JOURNAL OF PUBLIC HEALTH 18 (6): 427-438 DEC 2005 Message-ID: Cesar A. Macias-Chapula : E-mail : chapula at data.net.mx macias at bciencias.ucol.mx Title: Toward a model of communications in public health in Latin America and the Caribbean Author(s): Macias-Chapula CA Source: REVISTA PANAMERICANA DE SALUD PUBLICA-PAN AMERICAN JOURNAL OF PUBLIC HEALTH 18 (6): 427-438 DEC 2005 Document Type: Article Language: Spanish Cited References: 53 Abstract: Objective. So far, there have been no bibliometric or scientometric studies that make it possible to examine, with quantitative, retrospective, and comprehensive criteria, the scientific output on public health in Latin America and the Caribbean (LAC). Further, the weakness of the existing information systems makes it impossible to examine the relevance, quality, and impact of this scientific output, with a view to evaluating it in terms of societal needs and existing patterns of scientific communication. This article presents the results of a bibliographic analysis of the scientific output in the area of public health in Latin America and the Caribbean. The ultimate goal of the analysis is to build a model of scientific communication in this field, to help researchers, managers, and others working in the area of public health to make decisions and choose actions to take. Methods. We conducted a literature review in order to identify the distribution of publications on public health that were produced by LAC researchers and published in each of the LAC countries from 1980 through 2002. The review used the Literatura Latino-Americana e do Caribe em Saude Publica (LILACS-SP) (Latin American and Caribbean Literature on Public Health) bibliographic database. That database is operated by the Latin American and Caribbean Center on Health Sciences Information (BIREME), which is in Sao Paulo, Brazil. We processed the LILACS-SP data using two software packages, Microsoft Excel and Bibexcel, to obtain indicators of the scientific output, the type of document, the language, the number of authors for each publication, the thematic content, and the participating institutions. For the 1980-2002 period, there were 97 605 publications registered, from a total of 37 LAC countries. Results. For the analysis presented in this article, we limited the sample to the 8 countries in Latin America and the Caribbean that had at least 3 000 documents each registered in the LILACS-SP database over the 1980-2002 study period. In descending order of the number of publications registered, the 8 nations were: Argentina, Brazil, Chile, Colombia, Cuba, Mexico, Peru, and Venezuela. Those 8 countries were responsible for 83 054 publications (85.10% of the total of 97 605 registered documents produced by the 37 LAC countries). Of those 83 054 publications from the 8 countries, 56 253 of them (67.73%) were articles published in scientific journals and 24 488 were monographs (29.48%). The proportion of works produced by two or more coauthors was relatively high (56.48%). The 56 253 articles appeared in a total of 929 different journals. Of the 929 journals, 91 of them published at least 150 articles over the study period. In descending order, LAC journals with the largest number of articles on public health were: Revista de Saude Publica (Brazil); Cadernos de Saude Publica (Brazil); Revista Medica de Chile; Archivos Latinoamericanos de Nutricion (Venezuela); and Salud Publica de Mexico. The 91 journals that published at least 150 articles represented 29 different specialties. The most common of the specialties for the 91 journals were general medicine (18 journals) and pediatrics (10 journals). In descending order, the populations that the publications dealt with primarily were human beings in general, females, males, and adults; and, in descending order, a relatively small number of publications dealt with pregnant women and middle-aged or elderly persons. The topics most often covered in the publications were risk factors, health policy, and primary health care, as well as family doctors in the case of Cuba. Conclusions. This research produced a preliminary model of communications in public health in LAC countries that will hopefully help lay the groundwork for further research to develop a model of scientific communication in LAC nations. Addresses: Macias-Chapula CA (reprint author), Algeciras 43A,Col Insurgentes Mixcoac, Mexico City, DF 03920 Mexico Hosp Gen Mexico City, Direcc Invest, Mexico City, DF Mexico E-mail Addresses: chapula at data.net.mx Publisher: PAN AMERICAN HEALTH ORGANIZATION, 525 23RD ST NW, WASHINGTON, DC 20037 USA Subject Category: PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH; PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH IDS Number: 005AP ISSN: 1020-4989 REFERENCES : *GLOB FOR HLTH RES 10 90 REP HLTH RES 1 : 1999 *LANC ED LANCET 364 : 1555 2004 *OPS PROD CIENT SAL 1979 : 1989 ALLENDE JE MICROBIOLOGIA 13 : 127 1997 ARUNACHALAM S Tuberculosis research in India and China: From bibliometrics to research policy CURRENT SCIENCE 82 : 933 2002 ARUNACHALAN S Diabetes research in India and China today: From literature-based mapping to health-care policy CURRENT SCIENCE 82 : 1086 2002 AYALA FJ MICROBIOLOGIA 12 : 163 1996 BOOKSTEIN A TOWARDS A MULTIDISCIPLINARY BRADFORD LAW SCIENTOMETRICS 30 : 353 1994 BRACHORIQUELME RL Bibliometric repercussions of adopting English as the language of publication REVISTA DE INVESTIGACION CLINICA 49 : 369 1997 BRADFORD SC DOCUMENTATION : 1948 BROWN CG ARCH BIOL MED EXPT 24 : 37 1991 COIMBRA CEA CAD SAUDE PUBLICA 15 : 883 1999 DEASCENCIO MA PRODUCCION CIENTIFIC : 85 1992 DELOSRIOS R REV PANAM SALUD PUBL 5 : 309 1999 FIGUERA LE B OFICINA SANIT PANA 115 : 12 1993 FRENK J B OFICINA SANIT PANA 101 : 477 1986 GARCIA GG REV PANAM SALUD PUBL 9 : 406 2001 GARFIELD E B PAN AM HLTH ORG 29 : 87 1995 GIBBONS M NATURE S 402 : C82 1999 HORTON R The health (and wealth) of nations LANCET 359 : 993 2002 KRAUSKOPF M A scientometric radiography of Revista Medica de Chile REVISTA MEDICA DE CHILE 125 : 775 1997 LAMOINE VR ACTA CIENT VENEZ 43 : 321 1992 LANGER A SALUD PUBLICA MEXICO 33 : 410 1991 LEWINSON G SCI TECHN IND C 2002 : 2002 LONDONO JL Structured pluralism: Towards an innovative model for health system reform in Latin America HEALTH POLICY 41 : 1 1997 LOTUFO PA ARQ BRAS CARDIOL 76 : 419 2001 MACIASCHAPULA CA ANAL BIBLIOMETRICO P : 2003 MACIASCHAPULA CA AIDS in Haiti: a bibliometric analysis BULLETIN OF THE MEDICAL LIBRARY ASSOCIATION 88 : 56 2000 MACIASCHAPULA CA INVESTIGACION COMUNI : 55 2002 MACIASCHAPULA CA REDIAL 4 : 71 1994 MACIASCHAPULA CA REV ESP DOC CIENT 14 : 420 1991 MACIASCHAPULA CA Bibliometric and webometric analysis of health system reforms in Latin America and the Caribbean SCIENTOMETRICS 53 : 407 2002 MACIASCHAPULA CA Subject content analysis of AIDS literature, as produced in Latin America and the Caribbean SCIENTOMETRICS 46 : 563 1999 MACIASCHAPULA CA Bibliometric analysis of AIDS literature in Latin America and the Caribbean SCIENTOMETRICS 41 : 41 1998 MACIASCHAPULA CA Subject content of the Mexican production on health and the environment (1982-1993) SCIENTOMETRICS 38 : 295 1997 MACIASCHAPULA CA PRIMARY HEALTH-CARE IN MEXICO - A NON-ISI BIBLIOMETRIC ANALYSIS SCIENTOMETRICS 34 : 63 1995 MACIASCHAPULA CA NON-SCI SUBJECT VISIBILITY OF THE LATIN-AMERICAN SCIENTIFIC PRODUCTION IN THE HEALTH FIELD SCIENTOMETRICS 30 : 97 1994 MACIASCHAPULA CA PATTERNS OF SCIENTIFIC COMMUNICATION AMONG LATIN-AMERICAN COUNTRIES, IN THE FIELD OF MEDICAL-EDUCATION SCIENTOMETRICS 23 : 123 1992 MARTINEZ EV GAC MED MEX 136 : 319 2000 MASCASTELLA J MICROBIOL SEM 13 : 509 1997 NARROROBLES J SALUD PUBLICA MEXICO 33 : 605 1991 PASQUALINI CD The past and present of Medicina (Buenos Aires). MEDICINA-BUENOS AIRES 60 : 1 2000 PELLEGRINI A PRODUCCION CIENTIFIC : 19 1992 PELLEGRINI A PRODUCCION CIENTIFIC : 115 1992 PELLEGRINI A REV PANAM SALUD PUBL 1 : 23 1997 PORTA M The bibliographic ''impact factor'' of the Institute for Scientific Information: How relevant is it really for public health journals? Comments JOURNAL OF EPIDEMIOLOGY AND COMMUNITY HEALTH 50 : 606 1996 RODRIGUES PS Mapping cancer, cardiovascular and malaria research in Brazil BRAZILIAN JOURNAL OF MEDICAL AND BIOLOGICAL RESEARCH 33 : 853 2000 ROSELLI D MED EDUC 32 : 274 1998 SANBLAS G >From magic to science: a journey throughout Latin American medical mycology MEDICAL MYCOLOGY 38 : 1 2000 SANDOVAL A B UNESCO 28 : 1 1974 TAPIA L Authorship in the Chilean Journal of Otorhinolaryngology and Head and Neck Surgery. Analysis of three decades REVISTA MEDICA DE CHILE 130 : 773 2002 WEISINGER JR Latin American nephrology: Scientific production and impact of the publications KIDNEY INTERNATIONAL 56 : 1584 1999 WHITE F REV PANAM SALUD PUBL 12 : 165 2002 From garfield at CODEX.CIS.UPENN.EDU Mon Mar 27 14:18:06 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Mon, 27 Mar 2006 14:18:06 -0500 Subject: "How thematic maps can assist collection management: A qualitative assessment of Joumals' thematic focus " LIBRARY COLLECTIONS ACQUISITIONS & TECHNICAL SERVICES 29 (3): 295-306 2005 Message-ID: Fidelia Ibekwe-SanJuan : ibekwe at univ-lyon3.fr Title: How thematic maps can assist collection management: A qualitative assessment of Joumals' thematic focus Author(s): Ibekwe-SanJuan F Source: LIBRARY COLLECTIONS ACQUISITIONS & TECHNICAL SERVICES 29 (3): 295- 306 2005 Document Type: Article Language: English Cited References: 22 Times Cited: 0 Abstract: We present a method for mapping the content of a text collection. This method uses linguistic analysis to relate terms extracted from the texts and clusters them into thematic topics mapped onto a 2D space. While the graphic display of domain topics is useful for several information-driven tasks, the focus of the paper is more on the comparison of journal ranking by productivity (number of published papers in the collection) and by content representativity (ranking by number of terms and clusters). The results show that the two rankings are not identical, thus pointing to possible discrepancies between pure productivity and terminological density. (c) 2005 Elsevier Inc. All rights reserved. Addresses: Ibekwe-SanJuan F (reprint author), Enssib, URSIDOC, SII Res Lab, 17-21 Blvd 11 Novembre, Villeurbanne, F-69623 France Enssib, URSIDOC, SII Res Lab, Villeurbanne, F-69623 France Univ Lyon 3, Dept Informat Commun, Lyon, F-69008 France E-mail Addresses: ibekwe at univ-lyon3.fr Publisher: PERGAMON-ELSEVIER SCIENCE LTD, THE BOULEVARD, LANGFORD LANE, KIDLINGTON, OXFORD OX5 1GB, ENGLAND IDS Number: 999ED ISSN: 1464-9055 CITED REFERENCE: BAEZAYATES MODERN INFORM RETRIE : 117 1999 BOURIGAULT D RECENT ADV COMPUTATI 2 : 2001 CALLON M MAPPING DYNAMICS SCI : 1986 DEERWESTER S INDEXING BY LATENT SEMANTIC ANALYSIS JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 41 : 391 1990 FELLBAUM C WORDNET ELECT LEXICA : 1998 GILES CL CITATION INDEX J IMP HEARST M NATURAL LANGUAGE INF 7 : 333 1999 HEARST MA P 14 INT C COMP LING : 539 1992 IBEKWESANJUAN F 8 INT ISKO C U COLL : 41 2004 IBEKWESANJUAN F >From term variants to research topics KNOWLEDGE ORGANIZATION 29 : 181 2002 IBEKWESANJUAN F P 13 EUR C ART INT E : 170 1998 IBEKWESANJUAN F P INT C RECH INF ASS : 487 2004 JACQUEMIN C HDB COMPUTATIONAL LI : 599 2003 JARDINE N USE OF HIERARCHIC CLUSTERING IN INFORMATION RETRIEVAL INFORMATION STORAGE AND RETRIEVAL 7 : 217 1971 JONES KS AUTOMATIC KEYWORD CL : 1971 MORIN E COMPUT HUMANITIES 38 : 343 2004 SALTON G AUTOMATIC INFORM ORG : 1968 SCHIFFRIN R PUBLICATION NATL ACA 101 : 5183 2004 SMALL H Visualizing science by citation mapping JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 50 : 799 1999 WHITE HD BIBLIOMETRICS ANNUAL REVIEW OF INFORMATION SCIENCE AND TECHNOLOGY 24 : 119 1989 YEUNG KY Validating clustering for gene expression data BIOINFORMATICS 17 : 309 2001 ZAMIR O P 21 ANN INT ACM SIG : 46 1998 From garfield at CODEX.CIS.UPENN.EDU Mon Mar 27 15:26:11 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Mon, 27 Mar 2006 15:26:11 -0500 Subject: Wang P. and Bownas J "Research Classics and Citation Analysis" American Society for Information Science & Technology Proceedings of 2005 Annual Meeting Message-ID: Peiling Wang : peilingw at utk.edu Jennifer Bownas : jbownas at utk.edu FULL TEXT AVAILABLE AT : http://scholar.google.com/url? sa=U&q=http://web.utk.edu/~peilingw/asist05_202.pdf TITLE : Research Classics and Citation Analysis AUTHOR : Peiling Wang and Jennifer Bownas SOURCE : American Society for Information Science & Technology Proceedings of 2005 Annual Meeting E-mail addresses: {peilingw;jbownas}@utk.edu. ADDRESS : School of Information Sciences, University of Tennessee, Knoxville TN 37996 Introduction What are research classics? Research classics are works of extraordinary status in a field (Chubin, Porter, & Rossini, 1984). There are basically two ways to identify classic works: (1) experts' judgments; and (2) citation analysis. As an example of the former, many classics are selected by professors as course readings. Classic works may also be identified by citation counts. Citation Classics is a column in Current Contents? for the top cited works selected by the Institute for Scientific Information (ISI) (Garfield, 1987; 1989). As a quantitative measure, deciding on the threshold for classics seems a real challenge. Earlier, Price (1965) defines "4 percent of all papers appear to be "classics,' cited four or more times in a year." (p. 511) In a study of 23 physics classics published in the 1920s, Oppenheim & Renn (1978) found that they continued to receive between 13 and 114 citations in 1974 and 1975. Walstrom & Leonard (2000) used sustained citation counts, 10 per year over a 10 year period, to qualify superstar classics. In fact, many publications are never cited: 55% of scientific papers without a single citation within 5 years of publication and citation rates are much lower in social science and humanities (Hamilton, 1990). In this study, we examined 36 classics identified by 8 researchers cited in their research products (Wang & White, 1999). Using the threshold of 10 citations per year, 11 superstar classics (Table 1) were selected for indepth analysis: longitudinal citation plot, co-citation, recitation, and relationship of citing journals and classics. The nature of the superstars is further investigated using the data from Citation Classics and the newly launched ISI HighlyCited.com Website. The purpose of the study is to address the following research questions: 1. What do researchers mean when referring to classics in the context of use? In other words, what kinds of classics do they cite? 2. What are the characteristics of the 11 superstar classics? In other words, what are a. the longitudinal citation patterns? b. the co-citation patterns? c. the recitation patterns? d. the relationships between the citing journals and the journals that published the superstar classics? 3. What is the nature of the superstar classics? REFERENCES Aversa, E.S. (1984). Citation Patterns Of 400 Scientific Papers And Their Relationship To Literature Again. Unpublished doctoral dissertation, Drexel University, Philadelphia. Borgman, C.L. & Furner, J. (2002). Scholarly communication and bibliometrics. Annual Review of Information Science and Technology, Edited by Blaise Cronin, 37, 3-72. Brooks, T. A. (1986). Evidence of complex citer motivations. Journal of the American Society for Information Science, 37(1), 34-36. Cano, V., & Lind, N. C. (1991). Citation life cycles of ten citation classics. Scientometrics, 22(2), 297-312. Case, D. O., & Higgins, G. M. (2000). How can we investigate citation behavior?: A study of reasons for citing literature in communication. Journal of the American Society for Information Science, 51, 635-645. Chubin, D. E., Porter, A. L., & Rossini, F. A. 'Citation Classics' analysis: An approach to characterizing interdisciplinary research. Journal of the American Society for Information Science, 35(6), 360-368. Cronin, B (1984). The citation process: The role and significance of citations in scientific communication. London, Taylor Graham. Cronin, B., & Shaw, D. (2002).Identity-creators and image-makers: Using citation analysis and thick description to put authors in their place. Scientometrics, 54(1), 31-49. Diamond, A.M. (1989). The core journals of economics. Current Comments, January 2, 2-9. Garfield. E. (1987). Contemporary Classics in the social and behavioral sciences. Preface. NJ: ISI Press, xi-xv. Garfield. E. (1989). Citation Classics and citation behavior revisited. Current Comments, January 30 Hamilton, D.P. (1990). Publishing by ? and for? ? the numbers, Science 250, 1331-1332; Research papers: Who?s uncited now, Science 251, 25. McCain, K.W. (1990). Mapping authors in intellectual space: A technical overview. Journal of the American Society for Information Science, 41(6), 433-443. McCain, K.W. (1991). Mapping economics through the journal literature: An experiment in journal cocitation analysis. Journal of the American Society for Information Science, 42(2), 290-296. McCain, K.W. & Turner, K. (1989) Citation context analysis and aging patterns of journal articles in molecular genetics. Scientometrics 17:127-163. Merton, R.K. (1965) On the shoulders of giants: A Shandean postscript. New York: Harbourt Brace & World. Oppenheim, C. & Renn S.P. (1978) Highly cited old papers and the reasons why they continue to be cited. Journal of the American Society for Information Science, 29, 225-231. Price, Derek J. De Solla (1965). Networks of scientific papers. Science, New Series, 149(3683), 510-515. Page 10 Rice, R.E.C.L., Borgman, C.L., Reeves, B. (1988). Citation networks of communication journals, 1977-1985: Cliques and positions, citations made and citations received. Human communication research, 15, 256-283. Schelling, T. C. (1993) Bargaining - A citation-classic commentary on the strategy of conflict. Current Contents/Social & Behavioral Sciences, 6, 8. Shadish, W. R., Tolliver, D., Gray, M., & Sen Gupta, S. K. (1995). Author judgments about works they cite: Three studies from psychology journals. Social Studies of Science, 25, 47-498. Small, H. (1973) Co-citation in the scientific literature: A new measure of the relationship between two documents. Journal of American Society for Information Science, July-August, 265-269. Small, H. (1982). Citation context analysis. Progress in communication sciences volume III. 287-310 Walstrom & Leonard (2000). Citation classics from the information systems literature. Information & Management, 38, 59-72. Wang, P., & Soergel, D. (1998). A cognitive model of document use during a research project. Study I. Document selection. Journal of the American Society for Information Science, 49(2), 115-33. Wang, P., & White, M. D. (1999). A cognitive model of document use during a research project. Study II. Decisions at the reading and citing stages. Journal of the American Society for Information Science, 50(2), 98-114. White, H.D. (2000). Toward ego-centered citation analysis. In B. Cronin & H. B. Atkins (Eds.), The web of knowledge: A festschrift in honor of Eugene Garfield. (pp. 565 Medford, NJ: Information Today. White, H.D. (2001). Authors as citers over time. Journal of the American Society for Information Science and Technology, 52(2), 87-108. White, H.D. (2004). Citation analysis and discourse analysis revisited. Applied Linguistics, 25(1), 89-116. Wolfram, D. (2003). Applied informetrics for information retrieval research. Wetport, CT: Libraries unlimited. From garfield at CODEX.CIS.UPENN.EDU Mon Mar 27 15:34:01 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Mon, 27 Mar 2006 15:34:01 -0500 Subject: Georgas H, Cullars J "A citation study of the characteristics of the linguistics literature" COLLEGE & RESEARCH LIBRARIES 66 (6): 496-515 NOV 2005 Message-ID: Cullars J : jcullars at uic.edu Title: A citation study of the characteristics of the linguistics literature Author(s): Georgas H, Cullars J Source: COLLEGE & RESEARCH LIBRARIES 66 (6): 496-515 NOV 2005 Document Type: Article Language: English Cited References: 48 Times Cited: 0 Abstract: By analyzing the citation patterns of the linguistics literature, the authors provide a bibliometric description of the discipline that will help librarians who have reference, instruction, or collection development responsibilities in this area understand it better. One important aspect of such an understanding is determining where linguistics classifies within the humanities, the social sciences, and the sciences. Based on several of the citation patterns discovered, namely the importance of recent publications to the field, and the prominence of journals as a primary vehicle of scholarly communication, this analysis concludes that linguistics more closely resembles the disciplines of the social sciences. Addresses: Cullars J (reprint author), Univ Illinois, Chicago, IL 60607 USA Univ Illinois, Chicago, IL 60607 USA E-mail Addresses: jcullars at uic.edu Publisher: ASSOC COLL RESEARCH LIBRARIES, 50 E HURON ST, CHICAGO, IL 60611 USA IDS Number: 986PZ ISSN: 0010-0870 REFERENCES: ADAMS M LIT ENGLISH GUIDE LI : 248 2000 BALAY R GUIDE REFERENCE BOOK : 1996 BAUGHMAN JC STRUCTURAL-ANALYSIS OF LITERATURE OF SOCIOLOGY LIBRARY QUARTERLY 44 : 293 1974 BOWMAN M COLLECT BUILD 11 : 2 1991 BROADUS RN LITERATURE OF SOCIAL SCIENCES - SURVEY OF CITATION STUDIES INTERNATIONAL SOCIAL SCIENCE JOURNAL 23 : 236 1971 BROADUS RN USE OF PERIODICALS BY HUMANITIES SCHOLARS SERIALS LIBRARIAN 16 : 123 1989 BUCHANAN AL COMPARING MATERIALS USED IN PHILOSOPHY AND POLITICAL-SCIENCE DISSERTATIONS - A TECHNICAL NOTE BEHAVIORAL & SOCIAL SCIENCES LIBRARIAN 12 : 63 1993 BUDD J CHARACTERISTICS OF WRITTEN SCHOLARSHIP IN AMERICAN-LITERATURE - A CITATION STUDY LIBRARY & INFORMATION SCIENCE RESEARCH 8 : 189 1986 CALDER N MAGIC UNIVERSE OXFOR : 2003 CULLARS J CHARACTERISTICS OF THE MONOGRAPHIC LITERATURE OF BRITISH AND AMERICAN LITERARY-STUDIES COLLEGE & RESEARCH LIBRARIES 46 : 511 1985 CULLARS J CITATION CHARACTERISTICS OF MONOGRAPHS IN THE FINE-ARTS LIBRARY QUARTERLY 62 : 325 1992 CULLARS JM Citation characteristics of English-language monographs in philosophy LIBRARY & INFORMATION SCIENCE RESEARCH 20 : 41 1998 CULLARS JM Citation characteristics of French and German fine arts monographs LIBRARY QUARTERLY 66 : 138 1996 DELGADILLO R Future historians: Their quest for information COLLEGE & RESEARCH LIBRARIES 60 : 245 1999 DEMILLER AL LINGUISTICS GUIDE RE : 41 2000 FINEGAN E INTRO SCH MODERN LAN : 3 1992 FLETCHER J VIEW OF LITERATURE OF ECONOMICS JOURNAL OF DOCUMENTATION 28 : 283 1972 FROST CO USE OF CITATIONS IN LITERARY RESEARCH - PRELIMINARY CLASSIFICATION OF CITATION FUNCTIONS LIBRARY QUARTERLY 49 : 399 1979 GLANZEL W A bibliometric study of reference literature in the sciences and social sciences INFORMATION PROCESSING & MANAGEMENT 35 : 31 1999 GLANZEL W A bibliometric approach to social sciences, national research performances in 6 selected social science areas, 1990-1992 SCIENTOMETRICS 35 : 291 1996 HEINZKILL R CHARACTERISTICS OF REFERENCES IN SELECTED SCHOLARLY ENGLISH LITERARY JOURNALS LIBRARY QUARTERLY 50 : 352 1980 HERUBEL JVM COLLECTION MANAGEMEN 12 : 57 1990 HUNTER L COMMUNICATION 0401 : 2004 HUTCHINS WJ LANGUAGE BARRIER STU : 1971 KELLSEY C Global English in the humanities? A longitudinal citation study of foreign- language use by humanities scholars COLLEGE & RESEARCH LIBRARIES 65 : 194 2004 LINDHOLMROMANTSCHUK Y The role of monographs in scholarly communication: An empirical study of philosophy, sociology and economics JOURNAL OF DOCUMENTATION 52 : 389 1996 LINE MB INFORMATION USES AND NEEDS OF SOCIAL SCIENTISTS - OVERVIEW OF INFROSS ASLIB PROCEEDINGS 23 : 412 1971 LINE MB SOC SCI INFORM 1 : 67 1981 LOWE SM COLLECT BUILD 22 : 13 2003 MATTHEWS PH LINGUISTICS VERY SHO : 13 2003 MCELHINNY GENDER PUBLICATION C MCELHINNY B Gender, publication and citation in sociolinguistics and linguistic anthropology: The construction of a scholarly canon LANGUAGE IN SOCIETY 32 : 299 2003 NEDERHOF ASSESSING QUALITY SC NEDERHOF ASSESSING USEFULNESS NEDERHOF AJ INTERNATIONAL COMPARISON OF DEPARTMENTS RESEARCH PERFORMANCE IN THE HUMANITIES JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 43 : 249 1992 NEDERHOF AJ QUALITY JUDGMENTS OF JOURNALS AS INDICATORS OF RESEARCH PERFORMANCE IN THE HUMANITIES AND THE SOCIAL AND BEHAVIORAL-SCIENCES JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 42 : 332 1991 NEDERHOF AJ Assessing the quality of scholarly journals in Linguistics: An alternative to citation-based journal impact factors SCIENTOMETRICS 51 : 241 2001 NEDERHOF AJ ASSESSING THE USEFULNESS OF BIBLIOMETRIC INDICATORS FOR THE HUMANITIES AND THE SOCIAL AND BEHAVIORAL-SCIENCES - A COMPARATIVE-STUDY SCIENTOMETRICS 15 : 423 1989 NOVAK COMMUNICATION 0531 : 2001 NOVAK B COMMUNICATION 1120 : 2001 SEALEY A APPL LINGUISTICS SOC : 2004 SHARADA BA INDIAN LINGUISTICS 46 : 29 1988 STERN M CHARACTERISTICS OF THE LITERATURE OF LITERARY SCHOLARSHIP COLLEGE & RESEARCH LIBRARIES 44 : 199 1983 TANNEN D TALKING VOICES REPET : 197 1989 THOMPSON JW The death of the scholarly monograph in the humanities? Citation patterns in literary scholarship LIBRI 52 : 121 2002 WEBB WH SOURCES INFORMATION : 1986 YNGVE VH HARD SCI LINGUISTICS : R11 2004 ZWAAN RA SOME ASPECTS OF SCHOLARLY COMMUNICATION IN LINGUISTICS - AN EMPIRICAL-STUDY LANGUAGE 66 : 553 1990 From harnad at ECS.SOTON.AC.UK Mon Mar 27 18:47:21 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 28 Mar 2006 00:47:21 +0100 Subject: Manual Evaluation of Algorithm Performance on Identifying OA Message-ID: Previous AmSci Topic Thread: "Manual Evaluation of Algorithm Performance on Identifying OA" (Dec 2005) http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/5021.html Re: Antelman, K., Bakkalbasi, N., Goodman, D., Hajjem, C. and Harnad, S. (2005) Evaluation of Algorithm Performance on Identifying OA. Technical Report, North Carolina State University Libraries, North Carolina State University. http://eprints.ecs.soton.ac.uk/11689/ Hajjem, C., Harnad, S. and Gingras, Y. (2005) Ten-Year Cross-Disciplinary Comparison of the Growth of Open Access and How it Increases Research Citation Impact. IEEE Data Engineering Bulletin 28(4) pp. 39-47. http://eprints.ecs.soton.ac.uk/11688/ In an unpublished study, Antelman et al. (2005) hand-tested the accuracy of the algorithm that Hajjem et al.'s (2005) software robot used to identify Open Access (OA) and Non-Open-Access (NOA) articles in the ISI database. Antelman et al. found much lower accuracy (d' 0.98, bias 0.78, true OA 77%, False OA 41%), with their larger sample of nearly 600 (half OA, half NOA) in Biology (and even lower, near-chance performance in Sociology, sample size 600, d' 0.11, bias 0.99, true OA 53% false OA 49%) compared to Hajjem et al., who had found with their smaller Biology sample of 200 (d' 2.45, beta 0.52, true OA 93%, false OA 16%). Hajjem et al. have now re-done the hand-testing on a still larger sample (1000) in Biology, and we think we have identified the reason for the discrepancy, and demonstrated that Hajjem et al.'s original estimate of the robot's accuracy was closer to the correct one. The discrepancy was because Antelman et al. were hand-checking a sample other than the one the robot was sampling: The templates are the ISI articles. The ISI bibliographic data (author, title, etc.) for each article is first used to automatically trawl the web with search engines looking for hits, and then the robot applies its algorithm to the first 60 hits, calling the article "OA" if the algorithm thinks it has found at least one OA full-text among the 60 hits sampled, and NOA if it does not find one. Antelman et al. did not hand-check these same 60 hits for accuracy, because the hits themselves were not saved; the only thing recorded was the robot's verdict on whether a given article was OA or NOA. So Antelman et al. generated another sample -- with different search engines, on a different occasion -- for about 300 articles that the robot had previously identified as having an OA version in its sample, and 300 for which it had not found an OA version in its sample; Antelman et al.'s hand-testing found much lower accuracy. Hajjem et al.'s first test of the robot's accuracy made the very same mistake of hand-checking a new sample instead of saving the hits, and perhaps it yielded higher accuracy only because the time difference between the two samples was much smaller (but the search engines were again not the same ones used). Both accuracy hand-tests were based on incommensurable samples. Testing the robot's accuracy in this way is analogous to testing the accuracy of an instant blood test for the presence of a disease in a vast number of villages by testing a sample of 60 villagers in each (and declaring the disease to be present in the village (OA) if a positive case is detected in the sample of 60, NOA otherwise) and then testing the accuracy of the instant test against a reliable incubated test, but doing this by picking *another* sample of 60 from 100 of the villages that had previously been identified as "OA" based on the instant test and 100 that had been identified as "NOA." Clearly, to test the accuracy of the first, instant test, the second test ought to have been performed on the very same *individuals* on which the first test had been performed, not on another sample based only on the overall outcome of the first test, at the whole-village level. So when we hand-checked the actual hits (URLs) that the robot had identified as "OA" or "NOA" in our Biology sample of 1000, saving all the hits this time, the robot's accuracy was again much higher: d' 2.62, bias 0.68, true OA 93%, false OA 12%. All this merely concerned the robot's accuracy in detecting true OA. But our larger hand-checked sample now also allowed us to check whether the OA citation advantage (the ratio of the average citation counts for OA articles to the average citation counts for NOA articles in the same journal/issue) was an artifact of false OA: We accordingly had the robot's estimate of the OA Advantage of OA over NOA for this sample [(OA-NOA)/NOA = 70%], and we could now partition this into the ratio of the citation counts for true (93%) OA articles to the NOA articles (false NOA was very low, and would have worked against an OA advantage) versus the ratio of the citation counts for the false (12%) "OA" articles. The "false OA" advantage for this 12% of the articles was 33%, so there is definitely a false OA Advantage bias component in our results. However, the true OA advantage, for 93% of the articles, was 77%. So in fact, we are underestimating the OA advantage. As explained in previous postings, the purpose of the robot studies is not to get the most accurate possible estimate of the current percentage of OA in each field we study, nor even to get the most accurate possible estimate of the size of the OA citation Advantage. The advantage of a robot over much more accurate hand-testing is that we can look at a much larger sample, and faster -- indeed, we can test all of the articles in all the journals in each field in the ISI database, across years. Our interest at this point is in nothing more accurate than a rank-ordering of %OA as well as %OA citation Advantage across fields and years. We will nevertheless tighten the algorithm a little; the trick is not to make the algorithm so exacting for OA as to make it start producing substantially more false NOA errors, thereby weakening its overall accuracy for %OA as well as %OA advantage. Stevan Harnad From katy at INDIANA.EDU Tue Mar 28 00:28:21 2006 From: katy at INDIANA.EDU (Katy Borner) Date: Tue, 28 Mar 2006 00:28:21 -0500 Subject: Invitation: Opening Reception of the "Places & Spaces: Mapping Science" exhibit at NYPL on 4/3/06 In-Reply-To: <441D8425.1000805@indiana.edu> Message-ID: You are invited to join the Opening Reception of the Places & Spaces: Mapping Science exhibit at the Science, Industry and Business Library of The New York Public Library, New York, NY on Monday, April 3, 2006, 6:00-7:30pm. WHAT: Today, the word "science" encompasses myriad arenas of physical and abstract inquiry. This unique exhibition, at the Healy Hall in midtown Manhattan, uses innovative mapping techniques to physically show what and where science is today, how different branches of science relate to each other and where different branches of study are heading, where cutting edge science is erupting as archipelagos in the oceans of the yet unknown - and - how it all relates back to the physical centers of research. The world of science is turned into a navigable landscape. Modern mapping imagery has come a long way from Ptolemy. In this stimulating show compelling for all ages and backgrounds, audiences will both visually and tactilely uncover how contemporary scientific thought has expanded. Such visualization of scientific progress is approached through computer-generated relationships, featured on large panels as well through the collaboration of New York based artists W. Bradford Paley, Digital Image Design Incorporated and Columbia University and Ingo Gunther with renowned scientist from the field of scientonometrics: Eugene Garfield, Henry Small, Andr? Skupin, Steven A. Morris, Kevin W. Boyack and Dick Klavans. Scientists will be stimulated, students and teachers encouraged, and the general public fascinated by this multi-layered accessible approach to the worlds of modern scientific thought. WHEN: On Display April 3rd - August 31h, 2006 | Exhibit Opening is on April 3rd, 2006, 6-7:30pm WHERE: Healey Hall at the Science, Industry and Business Library of The New York Public Library 188 Madison Avenue, New York, NY 10016 (212) 592-7000 For information about this exhibit, please see: http://vw.indiana.edu/places&spaces/nypl http://www.nypl.org/research/calendar/exhib/sibl/uelistsibl.cfm (online exhibit available here) http://www.nypl.org/research/calendar/imagesexhib/places.pdf (exhibit brochure in PDF) Enjoy, Dr. Katy B?rner , Indiana University, Bloomington, IN > Deborah MacPherson , Accuracy&Aesthetics, Washington, D.C. > (Exhibit Curators) -- Katy Borner, Associate Professor Information Science & Cognitive Science Indiana University, SLIS 10th Street & Jordan Avenue Phone: (812) 855-3256 Fax: -6166 Main Library 019 E-mail: katy at indiana.edu Bloomington, IN 47405, USA WWW: ella.slis.indiana.edu/~katy Check out the new InfoVis Lab Gallery at http://ella.slis.indiana.edu/~katy/gallery/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From harnad at ECS.SOTON.AC.UK Tue Mar 28 06:19:27 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 28 Mar 2006 12:19:27 +0100 Subject: Future UK RAEs to be Metrics-Based Message-ID: This an anonymised exchange from a non-public list concerning scientometrics and the future of the UK Research Assessment Exercise. I think it has important general scientometric implications. By way of context: The RAE was an expensive, time-consuming submission/peer-re-evaluation exercise, performed every 4 years. It turned out a few simple metrics were highly correlated with its outcome. So it was proposed to scrap the expensive method in favour of just using the metrics. -- SH ---------- Forwarded message ---------- On Tue, 28 Mar 2006, [identity deleted] wrote: > At 8:34 am -0500 27/3/06, Stevan Harnad wrote: > >SH: Scrap the RAE make-work, by all means, but don't just rely on one > >metric! The whole point of metrics is to have many independent > >predictors, so as to account for as much as possible of the > >criterion variance: > > This seems extremely naive to me. All the proposed metrics I have > seen are *far* from independent - indeed they seem likely to be > strongly positively associated. that's fine. In multiple regression it is not necessary that each predictor variable be orthogonal; they need only predict a significant portion of the residual variance in the target (or "criterion") after the correlated portion has been partialled out. If you are trying to predict university performance and you have maths marks, english marks and letters of recommendation (quantified), it is not necessary, indeed not even desirable, that the correlation among the three predictors should be zero. That they are correlated shows that they are partially measuring the same thing. What is needed is that the three jointly, in a multilinear equation, should predict university performance better than any one of them alone. Their respective contributions to the variance can then be given a weight. The analogy is vectors, a linear combination of several of which may yield another, target vector. It need not be a linear combination of orthogonal vectors, just linearly independent ones. Three other points: (1) RCUK ranking itself is just a predictor, not the criterion that is being predicted and against which the predictor(s) need to be validated. The criterion is research performance/quality. Only metrics with face validity can be taken to be identical with the criterion, as opposed to mere predictors of it, and the RAE outcome is certainly not face-valid. (2) Given (1), it follows that the *extremely* high correlation between prior funding and RAE rank (0.98 was mentioned) is *not* a desirable thing. The predictive power of the RAE ranking needs to be increased, by adding more (semi-independent but not necessarily orthogonal) predictor metrics to a regression equation (such as funding, citations, downloads, co-citations, completed PhDs, and many other potential metrics that will emerge from an Open Access database and digital performance record-keeping CVs, customised for each discipline) rather than being replaced by a single one-dimensional predictor metric (prior funding) that happens to co-vary almost identically with the prior RAE outcome in many disciplines. (3) Validating predictor metrics against the target criterion is notoriously difficult when the criterion itself has no direct face-valid measure. (An example is the problem of validating IQ tests.) The solution is partly internal validation (validating multiple predictor metrics against one another) and partly calibration, which is the adjustment of the weight and number of the predictor metrics according to corrective feedback from their outcome: In the case of the RAE multiple regression equation, this could be done partly on the basis of the 4-year predictive power of metrics against their own later values, and partly against subjective peer rankings of departmental performance and quality as well as peer satisfaction ratings for the RAE outcomes themselves. (There may well be other validating methods.) > This sounds perilously close to what I used to read in the software > metrics literature, where attempts were made to capture 'complexity' > in order to predict the success or failure of software projects. > People there adopted a > measure-everything-you-can-think-of-and-hope-something-useful-pops-up > approach. The problem was that all the different metrics turned out > to be variants of 'size', and even together they did not enable good > prediction. It is conceivable but unlikely that all research performance predictor metrics turn out to be measuring the same thing, and that none of them contributes a separate independent component to the variance of the outcome; but I rather doubt it. At the risk of arousing other prejudices, I would make an analogy with psychometrics: Test of cognitive performance capacity (formerly called "IQ" tests) (maths, spatial, verbal, motor, musical, reasoning, etc.) are constructed and validated by devising test items and testing them first for reliability (i.e., how well they correlate with themselves on repeated administration) and then cross-correlation and external validation. The (empirical) result has been the emergence of one general or "G" factor for which the weight or "load" of some tests is greater than others, so that no single test measures it exactly, and hence a multiple regression battery, with each test weighted according to the amount of variance it accounts for, is preferable to relying on just a single test. And the outcome is that there turns out to be the one large underlying G factor, with a component in every one of the tests, plus a constellation of special factors, associated with special abilities supplementing the G factor, each adding a smaller but significant component to the variance too, but varying by individual and field in their predictive power. The controversy has been about whether the fact that the tests are validated on the basis of positive correlations among the items is the artifactual source of the positive manifold underlying G. I am not a statistician or a psychometrician, but I think the more competent, objective verdict (the one not driven by a-priori ideological views) has been that G is *not* an artifact of the selection for positive correlations, but a genuine empirical finding about a single general (indeed biological) factor underlying intelligence. I am not saying there will be a "G" underlying research performance! Just that the multilinear (and indeed nonlinear) regression method can be used to tease out the variance and the predictivity from a rich and diverse set of intercorrelated predictor metrics. (It can also sort out the duds, that are either redundant or predict nothing of interest at all.) > > SH: Metrics are trying to measure and evaluate research performance, > > I think you mean 'predict' - not the same thing at all They measure the predictor variable and try to predict the criterion variable. As such, they are meant to provide an objective (but validated) basis for evaluation. > >SH: not just to 2nd-guess the present RAE outcome, > >nor merely to ape existing funding levels. We need a rich multiple > >regression equation, with many weighted predictors, not just one > >redundant mirror image of existing funding! > > Well.... In fact 'existing funding' *may* actually be a good > predictor of whatever it is we want to predict (see [deleted]'s recent > posting)! To repeat: The RAE itself is a predictor, in want of validation. Prior funding correlates 0.98 with this predictor (in some fields, and is hence virtually identical with it), but is itself in want of validation. This high correlation with the actual RAE outcome is already rational grounds for scrapping the time-wasting and expensive ritual that is the present RAE, but it is certainly not grounds for scrapping other metrics that can and should be weighted components in the metric equation that replaces the current wasteful and redundant RAE. The metric predictors can then be enriched, cross-tested, and calibrated. (It is my understanding that RAE 2008 will consist of a double exercise: yet another iteration of the current ergonomically profligate RAE ritual plus a parallel metric exercise. I think they could safely scrap the ritual already, but the parallel testing of a rich battery of actual and potential metrics is an extremely good -- and economical -- idea.) > We can only test such hypotheses when we are clear what it > is we want to predict, and what we mean by 'accuracy' of prediction. In the first instance, in the decision about whether or not to scrap the expensive and inefficient current RAE ritual, it is sufficient to predict the current RAE outcome with metrics. In order to go on to test and strengthen the predictive power of that battery of metrics, they need to be enriched and diversified, internally validated and weighted against one another (and the prior RAE), and externally validated against the kinds of measure I mentioned (subjective peer evaluations, predictive power across time, perhaps other outcome metrics etc.) > Even if we knew this, I'm not sure the right data is available. But > in the absence of such a proper investigation, let's not pretend that > the answer is obvious, as you seem to be doing. The answer is obvious insofar as scrapping the prior RAE method is concerned, given the strong correlations. The answer is also obvious regarding the fact that multiple metrics are preferable to a single one. Ways of strengthening the predictive power of objective measures of research performance are practical and empirical matters we need to be analysing und upgrading continuously. Stevan Harnad From loet at LEYDESDORFF.NET Tue Mar 28 06:39:55 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Tue, 28 Mar 2006 13:39:55 +0200 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: > To repeat: The RAE itself is a predictor, in want of > validation. Prior funding correlates 0.98 with this predictor > (in some fields, and is hence virtually identical with it), > but is itself in want of validation. Do you wish to say that both the RAE and the multivariate regression method correlate highly with prior funding. Is the latter perhaps causal for research quality, in your opinion? The policy conclusion would then be that both indicators are very conservative. Perhaps, that is not a bad thing, but one may wish to state it straightforwardly. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ From harnad at ECS.SOTON.AC.UK Tue Mar 28 08:13:32 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 28 Mar 2006 08:13:32 -0500 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: The UK has a "dual" funding system: (1) conventional direct research grant applications, with peer review of competitive proposals (RCUK) and (2) top-sliced funding accorded to departments (not individuals) based on past departmental research performance (RAE). The RAE was a monstrously expensive and time-consuming exercise, with paper collection and submission of all kinds of performance markers, including 4 full-text papers, for peer-re-review by RAE panels. It turned out that the RAE's outcome -- each departmental RAE "rank" from 1 to 5*, with top-sliced funding given according to the rank and number of researchers submitted -- was highly correlated with total citation counts for the department's submitted researchers (r = .7 to .9+) and even more highly correlated with prior RCUK funding (.98). So RAE rank correlates highly with prior RCUK (and European) funding and almost as highly with citations (and with other metrics, such as number of doctorates accorded, etc.). The RAE rank is based on the data received and evaluated by the panel -- not through multiple regression, but through some sort of subjective weighting, including a "peer-re-review" of already published, already peer-reviewed articles (although I very much doubt many of them are actually read, the panels not being specific experts in their subject matter as the original journal peer-reviewers were meant to be -- it is far more likely that their ranking of the articles is based on the reputation of the journal in which they were published, and there is definitely pressure in the departments to preferentially submit articles that have been published in high-quality, high-impact journals). So what is counted explicitly is prior funding, doctorates, and a few other explicit measures; in addition, there is the "peer-re-review" -- whatever that amounts to -- which is no doubt *implicitly* influenced by journal reputations and impact factors. However, neither journal impact factors nor article/author citations are actually counted *explicitly* -- indeed it is explicitly forbidden to count citations for the RAE. That makes the high correlation of the RAE outcome with citation counts all the more remarkable -- and less remarkable than the even higher correlation with prior funding, which *is* counted explicitly. The multiple regression ("metric") method is not yet in use at all. It will now be tried out, in parallel with the next RAE (2008), which will be conducted in the usual way, but doing the metrics alongside. Prior funding counts are no doubt causal in the present RAE outcome (since they are explicitly counted), but that is not the same as saying that research funding is causal in generating research performance quality! (Funding is no doubt causal in being a necessary precondition for research quality, because without funding one cannot do research, but to what extent prior funding levels in and of themselves are causes of research quality variance over and above being a Matthew Effect or self-fulfilling prophecy is an empirical question about how good a predictor individual research-proposal peer- review is for allotting departmental top-sliced finding to reward and foster research performance. Hence the causality question is in a sense a question about the causal efficacy of UK's dual funding system itself, and the relative independence of its two components. For if they are indeed measuring and rewarding the very same thing, then RAE and the dual system may as well be scrapped, and the individual RCUK proposal funding with the redirected funds simply scaled up proportionately . I am not at all convinced that the dual system itself should be scrapped, however; just that the present costly and wasteful implementation of the RAE component should be replaced by metrics. And those metrics should certainly not be restricted to prior funding, even though it was so highly correlated with RAE ranking. It should be enriched by many other metric variables in a regression equation, composed and calibrated according to each discipline's peculiar profile as well as its internal and external validation results. And let us supplement conservative metrics with the many richer and more diverse ones that will be afforded by an online, open- access full-text corpus, citation-interlinked, tagged, and usage- monitored. Stevan Harnad On 28-Mar-06, at 6:39 AM, Loet Leydesdorff wrote >> To repeat: The RAE itself is a predictor, in want of >> validation. Prior funding correlates 0.98 with this predictor >> (in some fields, and is hence virtually identical with it), >> but is itself in want of validation. > > Do you wish to say that both the RAE and the multivariate > regression method > correlate highly with prior funding. Is the latter perhaps causal for > research quality, in your opinion? > > The policy conclusion would then be that both indicators are very > conservative. Perhaps, that is not a bad thing, but one may wish to > state it > straightforwardly. -------------- next part -------------- An HTML attachment was scrubbed... URL: From loet at LEYDESDORFF.NET Tue Mar 28 09:03:21 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Tue, 28 Mar 2006 16:03:21 +0200 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: <47EBE496-13DE-4FD2-9581-6D41ABD2F7B3@ecs.soton.ac.uk> Message-ID: The multiple regression ("metric") method is not yet in use at all. It will now be tried out, in parallel with the next RAE (2008), which will be conducted in the usual way, but doing the metrics alongside. Interesting! Do you expect the multiple regression model to correlate highly with the RAE? And if it will deviate, can you specify why? With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From harnad at ECS.SOTON.AC.UK Tue Mar 28 10:24:37 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 28 Mar 2006 16:24:37 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: On Tue, 28 Mar 2006, Loet Leydesdorff wrote: >> SH: The multiple regression ("metric") method is not yet in use at all. >> It will now be tried out, in parallel with the next RAE (2008), which >> will be conducted in the usual way, but doing the metrics alongside. http://www.hm-treasury.gov.uk/media/1E1/5E/bud06_science_332.pdf > Interesting! Do you expect the multiple regression model to correlate highly > with the RAE? And if it will deviate, can you specify why? It is already *known* that priori funding (which is explicitly counted) correlates very highly (.98), and citations (which are not explicitly counted) correlate highly (.7-.9) with the current RAE outcomes. So grants alone would predict all but 4% of the variance in the RAE rankings in many fields already. That is why the costly, time-consuming non-metric exercise is being dropped, and replaced entirely by metrics. But remember that the RAE itself is trying to measure/predict something, namely, research performance. And that prediction can certainly stand to be optimised. It is not optimised by simply relying on measures that duplicate the current RAE outcome, such as it is. Metrics should enrich and augment it. I cannot predict which way they will go in specific cases: I expect that the components and weightings of the metric multiple regression equation will vary from discipline to discipline, and will need to be calibrated. Some disciplines are more research-grant intensive than others, some journal-intensive, some book-intensive, some short-latency-impact-based, some long-: Parameters will have to be tweaked, and cross-validated against cross-years predictivity as well as peer ratings and feedback. And the full-text OA corpus will be a rapidly growing database, there to explore and data-mine for still further candidate metrics. Whilst saving a great deal of time and money, the new metric RAE can only increase its richness, sensitivity and diversity, compared to the old, bloated one. Stevan Harnad American Scientist Open Access Forum http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html Chaire de recherche du Canada Professor of Cognitive Science Ctr. de neuroscience de la cognition Dpt. Electronics & Computer Science Universit? du Qu?bec ? Montr?al University of Southampton Montr?al, Qu?bec Highfield, Southampton Canada H3C 3P8 SO17 1BJ United Kingdom http://www.crsc.uqam.ca/ http://www.ecs.soton.ac.uk/~harnad/ From loet at LEYDESDORFF.NET Tue Mar 28 11:15:53 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Tue, 28 Mar 2006 18:15:53 +0200 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: > I cannot predict which way they will go in specific > cases: I expect that the components and weightings of the > metric multiple regression equation will vary from discipline > to discipline, and will need to be calibrated. Some > disciplines are more research-grant intensive than others, > some journal-intensive, some book-intensive, some > short-latency-impact-based, some long-: Parameters will have > to be tweaked, and cross-validated against cross-years > predictivity as well as peer ratings and feedback. And the > full-text OA corpus will be a rapidly growing database, there > to explore and data-mine for still further candidate metrics. Dear Steven, I agree with this need for calibration, but it seems unlikely that this will be achieved within one RAE round. Thereafter, you have to wait four years for a second measurement point. :-) Furthermore, there is the problem of how to combine the multivariate regression with the time series perspective. The former will provide you with comparative statics, but not with insights in the dynamics of the system (autocorrelation problems!). It seems that you are formulating a research program rather than a policy measure. Perhaps, I lost you. I donwloaded the report of the UK government which you cited, but it is more on technology and innovation than on science. After a lot of the usual rethoric, they advise to develop "a metrics" without specifying anything. I lost your point. With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ From harnad at ECS.SOTON.AC.UK Tue Mar 28 11:41:34 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 28 Mar 2006 17:41:34 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: On Tue, 28 Mar 2006, Loet Leydesdorff wrote: > I agree with this need for calibration, but it seems unlikely that this will > be achieved within one RAE round. Thereafter, you have to wait four years > for a second measurement point. No, there are retro data available for the past RAEs as well. And the calibration is to improve the predictive power of the RAE. To merely duplicate its prior predictive power, the strongly correlated metrics are already enough. > Furthermore, there is the problem of how > to combine the multivariate regression with the time series perspective. The > former will provide you with comparative statics, but not with insights in > the dynamics of the system (autocorrelation problems!). It seems that you > are formulating a research program rather than a policy measure. There's definitely an emerging research programme there (Open Access Web Scientometrics) but there's no need for the results of the research to be in in order to conclude that the wasteful, inefficient non-metric components of the RAE should be replaced by the cheap, fast, efficient metrics that are already highly correlated with the outcome. The research programme is for *improving* on the prior outcomes. Bref: Scrapping the non-metric white-elephant in favour of existing metrics is the policy part; improving on existing metrics is the research part. > Perhaps, I lost you. I downloaded the report of the UK government which you > cited, but it is more on technology and innovation than on science. After a > lot of the usual rhetoric, they advise to develop "a metrics" without > specifying anything. The government's is just a policy document. Have a look at http://openaccess.eprints.org/index.php?/archives/75-guid.html for a summary and at the references cited in Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives: Improving the UK Research Assessment Exercise whilst making it cheaper and easier. Ariadne 35. http://www.ariadne.ac.uk/issue35/harnad/ for some of the correlational evidence. Stevan Harnad American Scientist Open Access Forum http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html Chaire de recherche du Canada Professor of Cognitive Science Ctr. de neuroscience de la cognition Dpt. Electronics & Computer Science Universit? du Qu?bec ? Montr?al University of Southampton Montr?al, Qu?bec Highfield, Southampton Canada H3C 3P8 SO17 1BJ United Kingdom http://www.crsc.uqam.ca/ http://www.ecs.soton.ac.uk/~harnad/ From ir at SOI.CITY.AC.UK Tue Mar 28 13:10:51 2006 From: ir at SOI.CITY.AC.UK (Ian Rowlands) Date: Tue, 28 Mar 2006 19:10:51 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: This is certainly an interesting technical argument, but I think we (the metrics / science policy community) ought to be trying to lead this debate to a more broadly socially useful conclusion. We know that targets are a very strong motivator and that they will have a profound effect on human behaviour. The argument so far (which I agree with) is that the RAE is a waste of money in that simple metrics could provide a solution which pretty well fits the rankings obtained - expensively - by committees. But the bigger question is what are we trying to achieve by the RAE or its successors? If it's simply research concentration, why not just let market forces rip? That would be both efficient and transparent. If the RAE is intended as a redistributive exercise, to address market failure, then we will need metrics (or fudges) that address that issue. Like how to deal with scholarship, editorial contributions, etc. But, fundamentally, what are we trying to achieve? Would not the RAE gain a modicum of respect if it justified itself in terms of changing academic behaviour towards some agreed community goals. I think this proposal is the worst of all worlds, it hardens the edges around the decisions that are made about a very complex set of scholarly communication behaviours and most definitely will not gain community respect, it does not address social value, the role of universities and where research should go (rather than where it's been). No good manager needs a load of trailing edge indicators to tell him or her who their best staff are. But they can see the value of setting aspirational targets that get the best from them by inspiring them to change their behaviour for the common good. We need new indicators that will enable policy makers to set socially desirable targets: should keep us going for a few years! Otherwise what is the point? Ian Rowlands Quoting Stevan Harnad : > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > On Tue, 28 Mar 2006, Loet Leydesdorff wrote: > > > I agree with this need for calibration, but it seems unlikely that this > will > > be achieved within one RAE round. Thereafter, you have to wait four years > > for a second measurement point. > > No, there are retro data available for the past RAEs as well. And the > calibration is to improve the predictive power of the RAE. To merely > duplicate its prior predictive power, the strongly correlated metrics are > already enough. > > > Furthermore, there is the problem of how > > to combine the multivariate regression with the time series perspective. > The > > former will provide you with comparative statics, but not with insights in > > the dynamics of the system (autocorrelation problems!). It seems that you > > are formulating a research program rather than a policy measure. > > There's definitely an emerging research programme there (Open Access Web > Scientometrics) but there's no need for the results of the research to > be in in order to conclude that the wasteful, inefficient non-metric > components of the RAE should be replaced by the cheap, fast, efficient > metrics that are already highly correlated with the outcome. The > research programme is for *improving* on the prior outcomes. > > Bref: Scrapping the non-metric white-elephant in favour of existing > metrics is the policy part; improving on existing metrics is the > research part. > > > Perhaps, I lost you. I downloaded the report of the UK government which you > > cited, but it is more on technology and innovation than on science. After a > > lot of the usual rhetoric, they advise to develop "a metrics" without > > specifying anything. > > The government's is just a policy document. Have a look at > > http://openaccess.eprints.org/index.php?/archives/75-guid.html > > for a summary and at the references cited in > > Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated > online RAE CVs Linked to University Eprint Archives: Improving the > UK Research Assessment Exercise whilst making it cheaper and easier. > Ariadne 35. http://www.ariadne.ac.uk/issue35/harnad/ > > for some of the correlational evidence. > > Stevan Harnad > American Scientist Open Access Forum > http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html > > Chaire de recherche du Canada Professor of Cognitive Science > Ctr. de neuroscience de la cognition Dpt. Electronics & Computer Science > Universit? du Qu?bec ? Montr?al University of Southampton > Montr?al, Qu?bec Highfield, Southampton > Canada H3C 3P8 SO17 1BJ United Kingdom > http://www.crsc.uqam.ca/ http://www.ecs.soton.ac.uk/~harnad/ > ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. From loet at LEYDESDORFF.NET Tue Mar 28 13:56:34 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Tue, 28 Mar 2006 20:56:34 +0200 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: > Bref: Scrapping the non-metric white-elephant in favour of > existing metrics is the policy part; improving on existing > metrics is the research part. Thus, you wish to make the RAE completely technocratic, while we know that we don't have reliable models for giving this strong type of policy advice. If I translate this for the Netherlands--having a dual system relying more on peer review than the UK--I can see the advantages, but also the disadvantages. For one, it might give my unit more money! With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ From garfield at CODEX.CIS.UPENN.EDU Tue Mar 28 14:47:06 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Tue, 28 Mar 2006 14:47:06 -0500 Subject: de la Guardia M "Editorial: Quantitative vibrational spectrometry in the 21st century" SPECTROSCOPY LETTERS 38 (6): 663-664 2005 Message-ID: M. de la Guardia: miguel.delaguardia at uv.es Title: Editorial: Quantitative vibrational spectrometry in the 21st century Author(s): de la Guardia M Source: SPECTROSCOPY LETTERS 38 (6): 663-664 2005 Document Type: Editorial Material Language: English Cited References: 0 Times Cited: 0 Publisher: TAYLOR & FRANCIS INC, 325 CHESTNUT ST, SUITE 800, PHILADELPHIA, PA 19106 USA Subject Category: SPECTROSCOPY IDS Number: 005JO ISSN: 0038-7010 ABSTRACT: The state of the art of research on vibrational spectrometry - based quantitative methodologies was evaluated from the literature compiled in Analytical Abstracts from 1980. Medium and near infrared, Raman spectrometry, and photoacoustic methods of analysis were considered. The evolution of the number of published papers, the distribution of the literature as a function of the different application fields in which the vibrational methods were employed, and a study of the impact on this area of chemometric and automation studies clearly shows that, from the 1990s until now, the importance of vibrational spectrometry in application analysis has grown to reach maturity. This field provides alternative methods for industrial, environmental, and food analysis and in clinical studies. The most active research groups on these subjects have been identified from their scientific production in the first years of this century and from the journals in which this research is commonly published. From PI at DB.DK Tue Mar 28 14:51:02 2006 From: PI at DB.DK (Peter Ingwersen) Date: Tue, 28 Mar 2006 21:51:02 +0200 Subject: Future UK RAEs to be Metrics-Based Message-ID: Dear Loet and Stevan et al. - I think we should stop this discussion now, prior to too many mails arriving into our boxes for no use. Please convey your discussion between you two. Loet is right in that the entire issue is political AND that the metrics we possess are not strong enough to substitute human assessemnts, e.g., of how the research is actually performed in the labs (good Lab. practice, etc.). Stevan might like the metrics, including fancy inlink analyses, not mentioned thus far, but: actually, the correlations referred to (published) cover only the top-ranked (and low-ranked) institutions in the RAE rankings - not really distinguishing between the large portion of mid-positioned research institutions in the UK. Hence, all this talk of substitution by metrics is beneficial/fair to some - not to the entire body of research. My best regards - Peter Ingwersen -----Original Message----- From: Loet Leydesdorff To: SIGMETRICS at LISTSERV.UTK.EDU Sent: 28-03-06 20:56 Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based > Bref: Scrapping the non-metric white-elephant in favour of > existing metrics is the policy part; improving on existing > metrics is the research part. Thus, you wish to make the RAE completely technocratic, while we know that we don't have reliable models for giving this strong type of policy advice. If I translate this for the Netherlands--having a dual system relying more on peer review than the UK--I can see the advantages, but also the disadvantages. For one, it might give my unit more money! With best wishes, Loet ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR), Kloveniersburgwal 48, 1012 CX Amsterdam. Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; loet at leydesdorff.net ; http://www.leydesdorff.net/ From ir at SOI.CITY.AC.UK Tue Mar 28 15:14:57 2006 From: ir at SOI.CITY.AC.UK (Ian Rowlands) Date: Tue, 28 Mar 2006 21:14:57 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: <991DCD1CCB9CD3118B38009027AA429E020BA336@loen.db.dk> Message-ID: Hi Peter No, I disagree. This is where the discussion starts. In the world of evidence-based policy, metricians hold the key. Metrics form targets, targets shape behaviour, behaviour determines outcomes. The metrics - outcomes relationship needs to be explored. Whether we like it or not, this is our responsibility, we are joined at the hip with the social responsibilities that come with judging others. Ian Quoting Peter Ingwersen : > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Dear Loet and Stevan et al. - I think we should stop this discussion now, > prior to too many mails arriving into our boxes for no use. Please convey > your discussion between you two. Loet is right in that the entire issue is > political AND that the metrics we possess are not strong enough to > substitute human assessemnts, e.g., of how the research is actually > performed in the labs (good Lab. practice, etc.). Stevan might like the > metrics, including fancy inlink analyses, not mentioned thus far, but: > actually, the correlations referred to (published) cover only the top-ranked > (and low-ranked) institutions in the RAE rankings - not really > distinguishing between the large portion of mid-positioned research > institutions in the UK. Hence, all this talk of substitution by metrics is > beneficial/fair to some - not to the entire body of research. My best > regards - Peter Ingwersen > > -----Original Message----- > From: Loet Leydesdorff > To: SIGMETRICS at LISTSERV.UTK.EDU > Sent: 28-03-06 20:56 > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > > Bref: Scrapping the non-metric white-elephant in favour of > > existing metrics is the policy part; improving on existing > > metrics is the research part. > > Thus, you wish to make the RAE completely technocratic, while we know > that > we don't have reliable models for giving this strong type of policy > advice. > If I translate this for the Netherlands--having a dual system relying > more > on peer review than the UK--I can see the advantages, but also the > disadvantages. For one, it might give my unit more money! > > With best wishes, > > > Loet > ________________________________ > Loet Leydesdorff > Amsterdam School of Communications Research (ASCoR), > Kloveniersburgwal 48, 1012 CX Amsterdam. > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > loet at leydesdorff.net ; http://www.leydesdorff.net/ > ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. From garfield at CODEX.CIS.UPENN.EDU Tue Mar 28 15:28:51 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Tue, 28 Mar 2006 15:28:51 -0500 Subject: Pecorari "Visible and occluded citation features in postgraduate second-language writing " English for Specific Purposes 25(1): 4-29, 2006 Message-ID: Diane Pecorari : E-mail Addresses: Diane.Pecorari at mdh.se Title: Visible and occluded citation features in postgraduate second- language writing Author(s): Pecorari D Source: ENGLISH FOR SPECIFIC PURPOSES 25 (1): 4-29 2006 Document Type: Article Language: English Cited References: 53 Times Cited: 0 Abstract: As novice members of their academic discourse communities, postgraduates face the challenge of learning to write in ways which will be judged as appropriate by those communities. Two resources in this effort are students' own observations of the features of published texts in their disciplines, and feedback on their texts from teachers and advisors. These resources depend, though, on the extent to which textual features can be observed. Swales [Swales, J. M. (1996). Occluded genres in the academy: The case of the submission letter. In E. Ventola & A. Mauranen (Eds.). Academic writing: intercultural and textual issues (pp. 45-58). Amsterdam: John Benjamins.] has noted the existence of occluded academic genres. The notion of occlusion is extended here to refer to the features of academic texts which are not ordinarily visible to the reader. One important area of occlusion is citation and, specifically, the relationship between a reference to a source and the source itself. This article reports the findings of an investigation into three visible and occluded features of postgraduate second-language writing. The novice writers in this study were found to respond to their disciplines' expectations in terms of the visible aspects of source use, but with regard to the occluded features their writing diverged considerably from received disciplinary norms. The findings also suggest that, with respect to disciplinary norms, a gap may exist between what is prescribed and what is practiced. (c) 2005 The American University. Published by Elsevier Ltd. All rights reserved. Addresses: Pecorari D (reprint author), Malardalen Univ, Dept Humanities, Vasteras, S-72123 Sweden Malardalen Univ, Dept Humanities, Vasteras, S-72123 Sweden E-mail Addresses: Diane.Pecorari at mdh.se Publisher: PERGAMON-ELSEVIER SCIENCE LTD, THE BOULEVARD, LANGFORD LANE, KIDLINGTON, OXFORD OX5 1GB, ENGLAND Subject Category: APPLIED LINGUISTICS IDS Number: 005FO ISSN: 0889-4906 EXCERPT FROM PAPER : DETAILS OF THE SOURCES: Given the preference of urban scholars to publish quickly and in journals, the expectation was that the biologists would cite more journal articles, and more recent ones, than the other writers. This expectation was met; the average age of the sources cited by the scientists was 9 years, but more than double that, ranging from 19 to 22 years, for the other fields (see Fig. 3). Research articles made up 85% of the sources cited by the biologists, but not more than 24% of those in the other fields (see Fig.4).determined actually to have been used. As Fig.3 shows, when the sources actually used are considered, the picture changes somewhat. The proportion of RAs among the scientists' sources is slightly lower - 80%. The sources used are also somewhat younger for all groups since, clearly, secondary sources have later publication dates than the primary sources they report. In (8a), for example, Ingrid cites a 1994 study by Kennard et al., but the similarity with Ferreira et al.'s slightly later paper (8b) suggests that Ingrid based her account on the latter. CITED REFERENCES : *AM PSYCH ASS PUBL MAN AM PSYCH AS : 2001 ACKER S THESIS SUPERVISION IN THE SOCIAL-SCIENCES - MANAGED OR NEGOTIATED HIGHER EDUCATION 28 : 483 1994 AGUILAR M J ENGLISH ACAD PURPO 3 : 55 2004 BARKS D LINKINGLIT PERSPECTI : 246 2001 BECHER T ACAD TRIBES TERRITOR : 2001 BELCHER D ACAD WRITING CONTEXT : 140 2001 BELCHER D ANN REV APPL LINGUIS 24 : 165 2004 BELCHER D ENGL SPECIF PURP 13 : 23 1994 BERG T CROSS LINGUISTIC COM : 1987 BERKENKOTTER C GENRE KNOWLEDGE DISC : 1995 BEST R ED CARE : 1983 BLOCH J LINKING LIT : 209 2001 BRENNER RP SOFT CLAY ENG : 159 1981 CHARLES M J ENGLISH ACAD PURPO 2 : 313 2003 COOPER DE METAPHOR : 1986 CROCKER J HERMES J LINGUISTICS 28 : 39 2002 CURRIE P J SECOND LANG WRIT 7 : 1 1998 DATTA A CURRENT TRENDS IN CANDIDA-ALBICANS RESEARCH ADVANCES IN MICROBIAL PHYSIOLOGY 30 : 53 1989 DEESE J THOUGHT SPEECH PSYCH : 1984 DEVITT AJ TEXTUAL DYNAMICS PRO : 336 1991 FERREIRA ME MAPPING LOCI CONTROLLING VERNALIZATION REQUIREMENT AND FLOWERING TIME IN BRASSICA-NAPUS THEORETICAL AND APPLIED GENETICS 90 : 727 1995 GIBALDI J MLA STYLE MANUAL GUI : 1998 GOSDEN H J ENGLISH ACAD PURPO 2 : 87 2003 GRIFFITH BC UNPUB STRUCTURE SOCI : 1983 GROOM N PATTERNS PERSPECTIVE : 14 2000 HOWARD RM PLAGIARISMS, AUTHORSHIPS, AND THE ACADEMIC DEATH-PENALTY COLLEGE ENGLISH 57 : 788 1995 HOWARD RM STANDING SHADOW GIAN : 1999 HULL G RETHINKING REMEDIATION - TOWARD A SOCIAL-COGNITIVE UNDERSTANDING OF PROBLEMATIC READING AND WRITING WRITTEN COMMUNICATION 6 : 139 1989 HYLAND K DISCIPLINARY DISCOUR : 2000 HYON S Beyond the research article: University faculty genres and EAP graduate preparation ENGLISH FOR SPECIFIC PURPOSES 23 : 233 2004 LAGERCRANTZ U Comparative mapping in Arabidopsis and Brassica, fine scale genome collinearity and congruence of genes controlling flowering time PLANT JOURNAL 9 : 13 1996 LAVE J SITUATED LEARNING LE : 1991 LUEBS M ENGL SPECIF PURP 17 : 67 1998 MATALENE C CONTRASTIVE RHETORIC, AN AMERICAN WRITING TEACHER IN CHINA COLLEGE ENGLISH 47 : 789 1985 MEGAHY T MANAGING PASTORAL CA : 26 1998 PECORARI D Good and original: Plagiarism and patchwriting in academic second-language writing JOURNAL OF SECOND LANGUAGE WRITING 12 : 317 2003 PECORARI D LINKING LITERACIES P : 229 2001 PETRIC B NOVELTY 11 : 4 2004 POPHAM JW ED EVALUATION : 1988 PRIOR PA WRITING DISCIPLINARI : 1998 SHATTUCKHUFNAGE.S LIMITED USE OF DISTINCTIVE FEATURES AND MARKEDNESS IN SPEECH PRODUCTION - EVIDENCE FROM SPEECH ERROR DATA JOURNAL OF VERBAL LEARNING AND VERBAL BEHAVIOR 18 : 41 1979 SHAW P REASONS FOR THE CORRELATION OF VOICE, TENSE, AND SENTENCE FUNCTION IN REPORTING VERBS APPLIED LINGUISTICS 13 : 302 1992 SHAW P J ENGLISH ACAD PURPO 2 : 343 2003 SWALES J WORLD ENGLISHES 16 : 373 1997 SWALES JM ACAD WRITING GRADUAT : 2004 SWALES JM ACAD WRITING INTERCU : 45 1996 SWALES JM ENGLISH TODAYS RES W : 2000 SWALES JM GENRE ANAL ENGLISH A : 1990 SWALES JM OTHER FLOORS OTHER V : 1998 SWALES JM RES GENRES EXPLORATI : 2004 TADROS A DATA DESCRIPTION DI : 98 1993 THOMPSON G EVALUATION IN THE REPORTING VERBS USED IN ACADEMIC PAPERS APPLIED LINGUISTICS 12 : 365 1991 WHITLEY R INTELLECTUAL SOCIAL : 1984 From garfield at CODEX.CIS.UPENN.EDU Tue Mar 28 16:30:04 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Tue, 28 Mar 2006 16:30:04 -0500 Subject: Noruzi A. "Web presence and impact factors for Middle-Eastern countries " ONLINE 30 (2): 22-+ MAR-APR 2006 Message-ID: Information on this paper : http://www.nouruzi.itgo.com/webometrics/Web_Impact_Factor.html Alireza Noruzi : anouruzi at yahoo.com Title: Web presence and impact factors for Middle-Eastern countries Author(s): Noruzi A Source: ONLINE 30 (2): 22-+ MAR-APR 2006 Document Type: Article Language: English Cited References: 13 Times Cited: 0 Abstract This study investigates the Web presence and Web Impact Factor (WIF) for country code top-level domains (ccTLDs) of Middle-Eastern countries, and sub-level domains (SLDs) related to education and academic institutions in these countries. Counts of links to the web sites of Middle-Eastern countries were calculated from the output of Yahoo search engine. In this study, we compute the WIF at two levels: top-level domains, and sub-level domains. The results show that the Middle-Eastern countries, apart from Turkey, Israel and Iran, have a low web presence. On the other hand, their web sites have a low inlink WIF. Specific features of sites may affect a country?s Web Impact Factor. For linguistic reasons, Middle-Eastern web sites (Persian, Kurdish, Turkish, Arabic, and Hebrew languages) may not receive and attract the attention that they deserve from the World Wide Web community. Addresses: Noruzi A (reprint author), Univ Teheran, Dept Lib & Informat Sci, Tehran, 14174 Iran Univ Teheran, Dept Lib & Informat Sci, Tehran, 14174 Iran E-mail Addresses: anouruzi at yahoo.com Publisher: ONLINE INC, 213 DANBURY RD, WILTON, CT 06897-4007 USA Subject Category: INFORMATION SCIENCE & LIBRARY SCIENCE IDS Number: 014FO ISSN: 0146-5422 CITED REFERENCES : ALMIND TC Informetric analyses on the World Wide Web: Methodological approaches to 'webometrics' JOURNAL OF DOCUMENTATION 53 : 404 1997 INGWERSEN P The calculation of Web impact factors JOURNAL OF DOCUMENTATION 54 : 236 1998 KOUSHA K J INFORM MANAGEMENT 1 : 13 2004 MOED HF The impact-factors debate: The ISI's uses and limits NATURE 415 : 731 2002 MUKHOPADHYAY P P NAT SEM INF SUPP R : 2004 NORUZI A ELECT LIB 23 : 2005 NORUZI A WEBOLOGY 2 : 2005 PENNOCK DM Winners don't take all: Characterizing the competition for links on the web PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA 99 : 5207 2002 RODRIGUEZ GJM REV ESPANOLA DOCUMEN 20 : 175 1997 SEGLEN PO Why the impact factor of journals should not be used for evaluating research BRITISH MEDICAL JOURNAL 314 : 498 1997 SMITH A Web impact factors for Australasian universities SCIENTOMETRICS 54 : 363 2002 SMITH AG WORLD LIB INF C 70 I : 2004 THELWALL M A comparison of sources of links for academic Web impact factor calculations JOURNAL OF DOCUMENTATION 58 : 66 2002 From garfield at CODEX.CIS.UPENN.EDU Tue Mar 28 16:35:44 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Tue, 28 Mar 2006 16:35:44 -0500 Subject: Barbui C, Cipriani A, Malvini L, Tansella M. "Validity of the impact factor of journals as a measure of randomized controlled trial quality " JOURNAL OF CLINICAL PSYCHIATRY 67 (1): 37-40 JAN 2006 Message-ID: E-mail Addresses: Corrado Barbui : corrado.barbui at univr.it Title: Validity of the impact factor of journals as a measure of randomized controlled trial quality Author(s): Barbui C, Cipriani A, Malvini L, Tansella M Source: JOURNAL OF CLINICAL PSYCHIATRY 67 (1): 37-40 JAN 2006 Document Type: Article Language: English Cited References: 20 Times Cited: 0 Abstract: Objective: To assess whether the impact factor, a measure of the frequency with which journal articles are cited in the scientific literature, is a proxy measure of the quality of articles reporting the results of randomized controlled trials. Method: The quality of trials included in an ongoing Cochrane review concerned with the antidepressant fluoxetine was assessed using the Cochrane Collaboration Depression, Anxiety, and Neurosis quality assessment instrument, the Jadad scale, and the quality criterion of the Cochrane Collaboration Handbook. Journal impact factors were extracted from the Journal Citation Report. Results: A total of 131 articles reported results from 132 clinical trials comparing fluoxetine with other antidepressants. The relationship between trial quality and the impact factor Of journals where these studies were published, stratified by period of publication, revealed that journals with impact factors above 4 points published only trials with above-average overall quality ratings, while journals with impact factors below 4 points published both high- and low-quality trials. The Jadad scale revealed similar quality in trials published in journals with high, medium, and low impact factors (Pearson chi(2) = 0.298, p = .861), and the quality criterion of the Cochrane Collaboration Handbook showed unclear randomization in the majority of trials and in all 15 trials published in high-impact factor journals (Pearson chi(2) = 4.678, p = .096). Conclusion: The impact factor of journals is not a valid measure of randomized controlled trial quality. KeyWords Plus: ANTIDEPRESSANT DRUG TRIALS; METAANALYSES; FLUOXETINE Addresses: Barbui C (reprint author), Univ Verona, Sect Psychiat & Clin Psychol, Dept Med & Publ Hlth, Policlin GB Rossi, Verona, I-37134 Italy Univ Verona, Sect Psychiat & Clin Psychol, Dept Med & Publ Hlth, Policlin GB Rossi, Verona, I-37134 Italy E-mail Addresses: corrado.barbui at univr.it Publisher: PHYSICIANS POSTGRADUATE PRESS, P O BOX 240008, MEMPHIS, TN 38124 USA Subject Category: PSYCHOLOGY, CLINICAL; PSYCHIATRY; PSYCHIATRY IDS Number: 009ZA ISSN: 0160-6689 CITED REFERENCES : *I SCI INF 19842003 I SCI INF ALTMAN DG Statistics notes - Treatment allocation in controlled trials: why randomise? BRITISH MEDICAL JOURNAL 318 : 1209 1999 AMIN M Impact factors: use and abuse MEDICINA-BUENOS AIRES 63 : 347 2003 BARBUI C Forty years of antidepressant drug trials ACTA PSYCHIATRICA SCANDINAVICA 104 : 92 2001 BARBUI C "Wish bias" in antidepressant drug trials? JOURNAL OF CLINICAL PSYCHOPHARMACOLOGY 24 : 126 2004 BLOCH S The Impact Factor: Time for change AUSTRALIAN AND NEW ZEALAND JOURNAL OF PSYCHIATRY 35 : 563 2001 BRAMBILLA P Side-effect profile of Fluoxetine in comparison with other SSRIs, tricyclic and newer antidepressants: A meta-analysis of clinical trial data PHARMACOPSYCHIATRY 38 : 69 2005 FURUKAWA TA Imputing response rates from means and standard deviations in meta-analyses INTERNATIONAL CLINICAL PSYCHOPHARMACOLOGY 20 : 49 2005 GARFIELD E Journal impact factor: a brief review CANADIAN MEDICAL ASSOCIATION JOURNAL 161 : 979 1999 JADAD M CONTROL CLIN TRIALS 17 : 1 1996 LEE KP Association of journal quality indicators with methodological quality of clinical research articles JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 287 : 2805 2002 LEFF D Making an impact: The rise of the impact factor as a measure of journal quality JOURNAL OF THE AMERICAN DIETETIC ASSOCIATION 105 : 29 2005 MOED HF Impact factors can mislead NATURE 381 : 186 1996 MOHER D Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? LANCET 352 : 609 1998 MONCRIEFF J INT J METH PSYCH RES 10 : 126 2001 PATTEN S International dosage differences in fluoxetine clinical trials CANADIAN JOURNAL OF PSYCHIATRY-REVUE CANADIENNE DE PSYCHIATRIE 50 : 31 2005 PORTA M Quality of impact factors of general medical journals - Quality matters - and the choice of indicator matters too BRITISH MEDICAL JOURNAL 326 : 931 2003 SACKETT D COCHRANE COLLABORATI : 1997 SAHA S Impact factor: a valid measure of journal quality? JOURNAL OF THE MEDICAL LIBRARY ASSOCIATION 91 : 42 2003 SEGLEN PO Why the impact factor of journals should not be used for evaluating research BRITISH MEDICAL JOURNAL 314 : 498 1997 From garfield at CODEX.CIS.UPENN.EDU Tue Mar 28 16:40:13 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Tue, 28 Mar 2006 16:40:13 -0500 Subject: Klass OS, Biham O, Levy M, Malcai O, Solomon S "The Forbes 400 and the Pareto wealth distribution " ECONOMICS LETTERS 90 (2): 290-295 FEB 2006 Message-ID: E-mail Addresses: mslm at mscc.huji.ac.il Title: The Forbes 400 and the Pareto wealth distribution Author(s): Klass OS, Biham O, Levy M, Malcai O, Solomon S Source: ECONOMICS LETTERS 90 (2): 290-295 FEB 2006 Document Type: Article Language: English Cited References: 19 Times Cited: 0 Abstract: Statistical regularities at the top end of the wealth distribution are examined using the Forbes 400 lists during 1988-2003. We find that the wealth is distributed according to a Pareto (power-law) distribution with an average exponent of alpha = 1.49. (c) 2005 Elsevier B.V. All rights reserved. Addresses: Levy M (reprint author), Hebrew Univ Jerusalem, Sch Business Adm, Jerusalem, IL-91905 Israel Hebrew Univ Jerusalem, Sch Business Adm, Jerusalem, IL-91905 Israel Hebrew Univ Jerusalem, Racah Inst Phys, Jerusalem, IL-91904 Israel ISI, Lagrange Lab Excellence Complex, Turin, Italy E-mail Addresses: mslm at mscc.huji.ac.il Publisher: ELSEVIER SCIENCE SA, PO BOX 564, 1001 LAUSANNE, SWITZERLAND Subject Category: ECONOMICS IDS Number: 012EQ ISSN: 0165-1765 CITED REFERENCES : ANDERSON PW EC EVOLVING COMPLEX 2 : 1997 ATKINSON AB DISTRIBUTION TOTAL W : 1978 ATKINSON AB The distribution of income in the UK and OECD countries in the twentieth century OXFORD REVIEW OF ECONOMIC POLICY 15 : 56 1999 KAHN LM Against the wind: Bargaining recentralisation and wage inequality in Norway 1987-91 ECONOMIC JOURNAL 108 : 603 1998 LAITNER J ECON J 11 : 691 2001 LEVY M EC EVOLVING COMPLEX 3 : 2005 LEVY M Dynamical explanation for the emergence of power law in a stock market model INTERNATIONAL JOURNAL OF MODERN PHYSICS C-PHYSICS AND COMPUTERS 7 : 65 1996 LEVY M Are rich people smarter? JOURNAL OF ECONOMIC THEORY 110 : 42 2003 LEVY M New evidence for the power-law distribution of wealth PHYSICA A 242 : 90 1997 PARETO V COURS EC POLITIQUE 2 : 1897 PERSKY J RETROSPECTIVES - PARETO LAW JOURNAL OF ECONOMIC PERSPECTIVES 6 : 181 1992 PIKETTY T 4631 CEPR : 2004 PIKETTY T Income inequality in France, 1901-1998 JOURNAL OF POLITICAL ECONOMY 111 : 1004 2003 SLOTTJE DJ STRUCTURE EARNINING : 1989 STECKEL RH Rising inequality: Trends in the distribution of wealth in industrializing New England JOURNAL OF ECONOMIC HISTORY 61 : 160 2001 STEINDL J RANDOM PROCESSES GRO : 1965 TAKAYASU H FRACTALS PHYS SCI : 1990 WOLFF EN TOP HEAVY STUDY INCR : 1995 ZIPF GK HUMAN BEHAV PRINCIPL : 1949 From dgoodman at PRINCETON.EDU Tue Mar 28 23:55:57 2006 From: dgoodman at PRINCETON.EDU (David Goodman) Date: Tue, 28 Mar 2006 23:55:57 -0500 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: <1143576897.442999417ea7a@fred.soi.city.ac.uk> Message-ID: Writing from a perspective where the RAE does not have any personal practical significance, I comment that such measures have a remarkable tendency to correlate with the preconceived rankings. For an area somewhat relevant to this list, journal selection and deselection, there are numerous factors besides IF. Librarians select among them to pretend to the faculty that objective decisions are being made. I have long experience in this art, and will be glad to demonstrate. Consider the importance of RAE to a department and an university. How could anyone in the UK could be both sufficently competent and unbiased to decide on the rankings of UK departments? Perhaps they should judge Japanese departments and vice versa. Dr. David Goodman Associate Professor Palmer School of Library and Information Science Long Island University and formerly Princeton University Library dgoodman at liu.edu dgoodman at princeton.edu ----- Original Message ----- From: Ian Rowlands Date: Tuesday, March 28, 2006 3:22 pm Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based To: SIGMETRICS at listserv.utk.edu > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Hi Peter > > No, I disagree. This is where the discussion starts. In the world of > evidence-based policy, metricians hold the key. Metrics form > targets, targets > shape behaviour, behaviour determines outcomes. The metrics - > outcomesrelationship needs to be explored. Whether we like it or > not, this is our > responsibility, we are joined at the hip with the social > responsibilities that > come with judging others. > > Ian > > Quoting Peter Ingwersen : > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > Dear Loet and Stevan et al. - I think we should stop this > discussion now, > > prior to too many mails arriving into our boxes for no use. > Please convey > > your discussion between you two. Loet is right in that the entire > issue is > > political AND that the metrics we possess are not strong enough to > > substitute human assessemnts, e.g., of how the research is actually > > performed in the labs (good Lab. practice, etc.). Stevan might > like the > > metrics, including fancy inlink analyses, not mentioned thus far, > but:> actually, the correlations referred to (published) cover only > the top-ranked > > (and low-ranked) institutions in the RAE rankings - not really > > distinguishing between the large portion of mid-positioned research > > institutions in the UK. Hence, all this talk of substitution by > metrics is > > beneficial/fair to some - not to the entire body of research. My > best> regards - Peter Ingwersen > > > > -----Original Message----- > > From: Loet Leydesdorff > > To: SIGMETRICS at LISTSERV.UTK.EDU > > Sent: 28-03-06 20:56 > > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based > > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > > Bref: Scrapping the non-metric white-elephant in favour of > > > existing metrics is the policy part; improving on existing > > > metrics is the research part. > > > > Thus, you wish to make the RAE completely technocratic, while we > know> that > > we don't have reliable models for giving this strong type of policy > > advice. > > If I translate this for the Netherlands--having a dual system > relying> more > > on peer review than the UK--I can see the advantages, but also the > > disadvantages. For one, it might give my unit more money! > > > > With best wishes, > > > > > > Loet > > ________________________________ > > Loet Leydesdorff > > Amsterdam School of Communications Research (ASCoR), > > Kloveniersburgwal 48, 1012 CX Amsterdam. > > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; > > loet at leydesdorff.net ; http://www.leydesdorff.net/ > > > > > > > ---------------------------------------------------------------- > This message was sent using IMP, the Internet Messaging Program. > From dgoodman at PRINCETON.EDU Thu Mar 30 00:43:55 2006 From: dgoodman at PRINCETON.EDU (David Goodman) Date: Thu, 30 Mar 2006 00:43:55 -0500 Subject: Manual Evaluation of Algorithm Performanceon Identifying OA Message-ID: The three of us are delighted that Stevan and Chawki are continuing work on this topic. Not having seen the full actual data from their latest study, we cannot fully evaluate it. However, in our view, their present explanation and analysis is very reasonable. We agree that there is yet more to be done, and regret that we are not now able to participate. David Goodman, Nisa Bakkalbasi and Kristin Antelman posted by David Goodman for our group. Associate Professor Palmer School of Library and Information Science Long Island University dgoodman at liu.edu -----Original Message----- From: Stevan Harnad Sent: Mon 3/27/2006 6:47 PM Cc: SIGMETRICS at LISTSERV.UTK.EDU Subject: Re: Manual Evaluation of Algorithm Performanceon Identifying OA Previous Topic Thread: "Manual Evaluation of Algorithm Performance on Identifying OA" (Dec 2005) http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/5021.html Re: Antelman, K., Bakkalbasi, N., Goodman, D., Hajjem, C. and Harnad, S. (2005) Evaluation of Algorithm Performance on Identifying OA. Technical Report, North Carolina State University Libraries, North Carolina State University. http://eprints.ecs.soton.ac.uk/11689/ Hajjem, C., Harnad, S. and Gingras, Y. (2005) Ten-Year Cross-Disciplinary Comparison of the Growth of Open Access and How it Increases Research Citation Impact. IEEE Data Engineering Bulletin 28(4) pp. 39-47. http://eprints.ecs.soton.ac.uk/11688/ In an unpublished study, Antelman et al. (2005) hand-tested the accuracy of the algorithm that Hajjem et al.'s (2005) software robot used to identify Open Access (OA) and Non-Open-Access (NOA) articles in the ISI database. Antelman et al. found much lower accuracy (d' 0.98, bias 0.78, true OA 77%, False OA 41%), with their larger sample of nearly 600 (half OA, half NOA) in Biology (and even lower, near-chance performance in Sociology, sample size 600, d' 0.11, bias 0.99, true OA 53% false OA 49%) compared to Hajjem et al., who had found with their smaller Biology sample of 200 (d' 2.45, beta 0.52, true OA 93%, false OA 16%). Hajjem et al. have now re-done the hand-testing on a still larger sample (1000) in Biology, and we think we have identified the reason for the discrepancy, and demonstrated that Hajjem et al.'s original estimate of the robot's accuracy was closer to the correct one. The discrepancy was because Antelman et al. were hand-checking a sample other than the one the robot was sampling: The templates are the ISI articles. The ISI bibliographic data (author, title, etc.) for each article is first used to automatically trawl the web with search engines looking for hits, and then the robot applies its algorithm to the first 60 hits, calling the article "OA" if the algorithm thinks it has found at least one OA full-text among the 60 hits sampled, and NOA if it does not find one. Antelman et al. did not hand-check these same 60 hits for accuracy, because the hits themselves were not saved; the only thing recorded was the robot's verdict on whether a given article was OA or NOA. So Antelman et al. generated another sample -- with different search engines, on a different occasion -- for about 300 articles that the robot had previously identified as having an OA version in its sample, and 300 for which it had not found an OA version in its sample; Antelman et al.'s hand-testing found much lower accuracy. Hajjem et al.'s first test of the robot's accuracy made the very same mistake of hand-checking a new sample instead of saving the hits, and perhaps it yielded higher accuracy only because the time difference between the two samples was much smaller (but the search engines were again not the same ones used). Both accuracy hand-tests were based on incommensurable samples. Testing the robot's accuracy in this way is analogous to testing the accuracy of an instant blood test for the presence of a disease in a vast number of villages by testing a sample of 60 villagers in each (and declaring the disease to be present in the village (OA) if a positive case is detected in the sample of 60, NOA otherwise) and then testing the accuracy of the instant test against a reliable incubated test, but doing this by picking *another* sample of 60 from 100 of the villages that had previously been identified as "OA" based on the instant test and 100 that had been identified as "NOA." Clearly, to test the accuracy of the first, instant test, the second test ought to have been performed on the very same *individuals* on which the first test had been performed, not on another sample based only on the overall outcome of the first test, at the whole-village level. So when we hand-checked the actual hits (URLs) that the robot had identified as "OA" or "NOA" in our Biology sample of 1000, saving all the hits this time, the robot's accuracy was again much higher: d' 2.62, bias 0.68, true OA 93%, false OA 12%. All this merely concerned the robot's accuracy in detecting true OA. But our larger hand-checked sample now also allowed us to check whether the OA citation advantage (the ratio of the average citation counts for OA articles to the average citation counts for NOA articles in the same journal/issue) was an artifact of false OA: We accordingly had the robot's estimate of the OA Advantage of OA over NOA for this sample [(OA-NOA)/NOA = 70%], and we could now partition this into the ratio of the citation counts for true (93%) OA articles to the NOA articles (false NOA was very low, and would have worked against an OA advantage) versus the ratio of the citation counts for the false (12%) "OA" articles. The "false OA" advantage for this 12% of the articles was 33%, so there is definitely a false OA Advantage bias component in our results. However, the true OA advantage, for 93% of the articles, was 77%. So in fact, we are underestimating the OA advantage. As explained in previous postings, the purpose of the robot studies is not to get the most accurate possible estimate of the current percentage of OA in each field we study, nor even to get the most accurate possible estimate of the size of the OA citation Advantage. The advantage of a robot over much more accurate hand-testing is that we can look at a much larger sample, and faster -- indeed, we can test all of the articles in all the journals in each field in the ISI database, across years. Our interest at this point is in nothing more accurate than a rank-ordering of %OA as well as %OA citation Advantage across fields and years. We will nevertheless tighten the algorithm a little; the trick is not to make the algorithm so exacting for OA as to make it start producing substantially more false NOA errors, thereby weakening its overall accuracy for %OA as well as %OA advantage. Stevan Harnad From jonathan at LEVITT.NET Thu Mar 30 10:30:16 2006 From: jonathan at LEVITT.NET (Jonathan Levitt) Date: Thu, 30 Mar 2006 16:30:16 +0100 Subject: Future UK RAEs to be Metrics-Based Message-ID: Hi, As a student in London, I have found the debate on the UK RAE very interesting. My suggestion is that it is important to consider, not only which metric to apply, but also other issues such as: 1) Whom to rate: Currently departments are rated in the RAE. Is it a good idea for the RAE to allocate funding to departments rather than to the researcher or the research team? Currently, no funding would be allocated to an exceptional researcher or research team at a lowly rated department. 2) How often to rate: Currently the rating takes place every six years. Is it a good idea to make funding decisions so infrequently? I understand that other funding bodies make these decisions on an annual basis. 3) How accessible to make the results: The results of the RAE exercises are published on the Internet. Is it a good idea to make these ratings so widely accessible? Before I applied to be a research student in London I downloaded the RAE ratings of the departments in my field in London and based my decision on where to study on the RAE ratings. Best regards, Jonathan. ----- Original Message ----- From: "David Goodman" To: Sent: Wednesday, March 29, 2006 5:55 AM Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Writing from a perspective where the RAE does not have > any personal practical significance, I comment that such > measures have a remarkable tendency to > correlate with the preconceived rankings. > > For an area somewhat relevant to this list, > journal selection and deselection, there are > numerous factors besides IF. Librarians select among > them to pretend to the faculty that objective decisions are > being made. I have long experience in this art, and > will be glad to demonstrate. > > Consider the importance of RAE to a department and an > university. How could anyone in the UK could be > both sufficently competent and unbiased > to decide on the rankings of UK departments? Perhaps they > should judge Japanese departments and vice versa. > > Dr. David Goodman > Associate Professor > Palmer School of Library and Information Science > Long Island University > and formerly > Princeton University Library > > dgoodman at liu.edu > dgoodman at princeton.edu > > ----- Original Message ----- > From: Ian Rowlands > Date: Tuesday, March 28, 2006 3:22 pm > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based > To: SIGMETRICS at listserv.utk.edu > >> Adminstrative info for SIGMETRICS (for example unsubscribe): >> http://web.utk.edu/~gwhitney/sigmetrics.html >> >> Hi Peter >> >> No, I disagree. This is where the discussion starts. In the world of >> evidence-based policy, metricians hold the key. Metrics form >> targets, targets >> shape behaviour, behaviour determines outcomes. The metrics - >> outcomesrelationship needs to be explored. Whether we like it or >> not, this is our >> responsibility, we are joined at the hip with the social >> responsibilities that >> come with judging others. >> >> Ian >> >> Quoting Peter Ingwersen : >> >> > Adminstrative info for SIGMETRICS (for example unsubscribe): >> > http://web.utk.edu/~gwhitney/sigmetrics.html >> > >> > Dear Loet and Stevan et al. - I think we should stop this >> discussion now, >> > prior to too many mails arriving into our boxes for no use. >> Please convey >> > your discussion between you two. Loet is right in that the entire >> issue is >> > political AND that the metrics we possess are not strong enough to >> > substitute human assessemnts, e.g., of how the research is actually >> > performed in the labs (good Lab. practice, etc.). Stevan might >> like the >> > metrics, including fancy inlink analyses, not mentioned thus far, >> but:> actually, the correlations referred to (published) cover only >> the top-ranked >> > (and low-ranked) institutions in the RAE rankings - not really >> > distinguishing between the large portion of mid-positioned research >> > institutions in the UK. Hence, all this talk of substitution by >> metrics is >> > beneficial/fair to some - not to the entire body of research. My >> best> regards - Peter Ingwersen >> > >> > -----Original Message----- >> > From: Loet Leydesdorff >> > To: SIGMETRICS at LISTSERV.UTK.EDU >> > Sent: 28-03-06 20:56 >> > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based >> > >> > Adminstrative info for SIGMETRICS (for example unsubscribe): >> > http://web.utk.edu/~gwhitney/sigmetrics.html >> > >> > > Bref: Scrapping the non-metric white-elephant in favour of >> > > existing metrics is the policy part; improving on existing >> > > metrics is the research part. >> > >> > Thus, you wish to make the RAE completely technocratic, while we >> know> that >> > we don't have reliable models for giving this strong type of policy >> > advice. >> > If I translate this for the Netherlands--having a dual system >> relying> more >> > on peer review than the UK--I can see the advantages, but also the >> > disadvantages. For one, it might give my unit more money! >> > >> > With best wishes, >> > >> > >> > Loet >> > ________________________________ >> > Loet Leydesdorff >> > Amsterdam School of Communications Research (ASCoR), >> > Kloveniersburgwal 48, 1012 CX Amsterdam. >> > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681; >> > loet at leydesdorff.net ; http://www.leydesdorff.net/ From jonathan at LEVITT.NET Thu Mar 30 11:45:44 2006 From: jonathan at LEVITT.NET (Jonathan Levitt) Date: Thu, 30 Mar 2006 17:45:44 +0100 Subject: Future UK RAEs to be Metrics-Based Message-ID: Hi, As a student in London, I have found the debate on the UK RAE very interesting. My suggestion is that it is important to consider, not only which metric to apply, but also other issues such as: 1) Whom to rate: Currently departments are rated in the RAE. Is it a good idea for the RAE to allocate funding to departments rather than to the researcher or the research team? Currently, no funding would be allocated to an exceptional researcher or research team at a lowly rated department. 2) How often to rate: Currently the rating takes place every six years. Is it a good idea to make funding decisions so infrequently? I understand that other funding bodies make these decisions on an annual basis. 3) How accessible to make the results: The results of the RAE exercises are published on the Internet. Is it a good idea to make these ratings so widely accessible? Before I applied to be a research student in London I downloaded the RAE ratings of the departments in my field in London and based my decision on where to study on the RAE ratings. Best regards, Jonathan. ----- Original Message ----- From: "Stevan Harnad" To: Sent: Thursday, March 23, 2006 3:47 PM Subject: [SIGMETRICS] Future UK RAEs to be Metrics-Based > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > ** Apologies for Cross-Posting ** > > Don't say we didn't tell you so! The wasteful, time-consuming RAE will > be replaced by metrics, chief among them citation impact, which already > correlated with and predicted the RAE outcome anyway, without being > explicitly counted. Now it can be explicitly counted (along with other > powerful new metrics) and all the rest of the ritualistic time-wasting > can be abandoned, without ceremony. This is a great boost for > institutional self-archiving in OA Institutional Repositories, not only > as the obvious, optimal means of submission, but as the means of > maximising research impact: http://irra.eprints.org/software/bronze/ > (I hope RCUK is listening!): > > "Research exercise to be scrapped" > Donald MacLeod, Guardian Wednesday March 22, 2006 > http://education.guardian.co.uk/RAE/story/0,,1737082,00.html > > Cf: Harnad, S. (2001) Why I think that research access, impact and > assessment > are linked. Times Higher Education Supplement 1487: p. 16. > http://www.thes.co.uk/search/story.aspx?story_id=74889 > http://cogprints.org/1683/ > > Harnad, S. (2003) Why I believe that all UK research output > should be online. > Times Higher Education Supplement. Friday, June 6 2003. > http://www.thes.co.uk/search/story.aspx?story_id=92599 > http://eprints.ecs.soton.ac.uk/7728/ > > Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated > online RAE CVs Linked to University Eprint Archives: Improving > the UK Research Assessment Exercise whilst making it cheaper and > easier. Ariadne. > http://www.ariadne.ac.uk/issue35/harnad/ From notsjb at LSU.EDU Thu Mar 30 12:53:54 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Thu, 30 Mar 2006 11:53:54 -0600 Subject: Future UK RAEs to be Metrics-Based Message-ID: Speaking as a capitalist pig, the entire RAE system is just another example of socialists hoisting themselves on their own petards. Point 1 below contains the essence of the problem. The US has done pioneering work on the evaluation of research-doctorate programs but was never silly enough to allocate research resources on the basis of it. Luckily because these evaluations were usually screwed up in some way. Allocation of research resources was always done on a project-by-project basis by the NSF, NIH, and others, with experts in the fields evaluating individual research proposals. The Europeans have a tendency to overplan everything with disastrous consequences--the disaster in Eastern Europe just being the latest example of it. SB Jonathan Levitt @LISTSERV.UTK.EDU> on 03/30/2006 10:45:44 AM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based Hi, As a student in London, I have found the debate on the UK RAE very interesting. My suggestion is that it is important to consider, not only which metric to apply, but also other issues such as: 1) Whom to rate: Currently departments are rated in the RAE. Is it a good idea for the RAE to allocate funding to departments rather than to the researcher or the research team? Currently, no funding would be allocated to an exceptional researcher or research team at a lowly rated department. 2) How often to rate: Currently the rating takes place every six years. Is it a good idea to make funding decisions so infrequently? I understand that other funding bodies make these decisions on an annual basis. 3) How accessible to make the results: The results of the RAE exercises are published on the Internet. Is it a good idea to make these ratings so widely accessible? Before I applied to be a research student in London I downloaded the RAE ratings of the departments in my field in London and based my decision on where to study on the RAE ratings. Best regards, Jonathan. ----- Original Message ----- From: "Stevan Harnad" To: Sent: Thursday, March 23, 2006 3:47 PM Subject: [SIGMETRICS] Future UK RAEs to be Metrics-Based > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > ** Apologies for Cross-Posting ** > > Don't say we didn't tell you so! The wasteful, time-consuming RAE will > be replaced by metrics, chief among them citation impact, which already > correlated with and predicted the RAE outcome anyway, without being > explicitly counted. Now it can be explicitly counted (along with other > powerful new metrics) and all the rest of the ritualistic time-wasting > can be abandoned, without ceremony. This is a great boost for > institutional self-archiving in OA Institutional Repositories, not only > as the obvious, optimal means of submission, but as the means of > maximising research impact: http://irra.eprints.org/software/bronze/ > (I hope RCUK is listening!): > > "Research exercise to be scrapped" > Donald MacLeod, Guardian Wednesday March 22, 2006 > http://education.guardian.co.uk/RAE/story/0,,1737082,00.html > > Cf: Harnad, S. (2001) Why I think that research access, impact and > assessment > are linked. Times Higher Education Supplement 1487: p. 16. > http://www.thes.co.uk/search/story.aspx?story_id=74889 > http://cogprints.org/1683/ > > Harnad, S. (2003) Why I believe that all UK research output > should be online. > Times Higher Education Supplement. Friday, June 6 2003. > http://www.thes.co.uk/search/story.aspx?story_id=92599 > http://eprints.ecs.soton.ac.uk/7728/ > > Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated > online RAE CVs Linked to University Eprint Archives: Improving > the UK Research Assessment Exercise whilst making it cheaper and > easier. Ariadne. > http://www.ariadne.ac.uk/issue35/harnad/ From garfield at CODEX.CIS.UPENN.EDU Thu Mar 30 14:02:42 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Thu, 30 Mar 2006 14:02:42 -0500 Subject: McBride R. "Impact factors for the Journal of Teaching in Physical Education - What are they and are they important? " JOURNAL OF TEACHING IN PHYSICAL EDUCATION 25 (1): 3-8 JAN 2006 Message-ID: Ron McBride: rmac at tamu.edu Title: Impact factors for the Journal of Teaching in Physical Education - What are they and are they important? Author(s): McBride R Source: JOURNAL OF TEACHING IN PHYSICAL EDUCATION 25 (1): 3-8 JAN 2006 Document Type: Editorial Material Language: English Cited References: 11 Times Cited: 0 Addresses: McBride R (reprint author), Texas A&M Univ, Dept Hlth & Kinesiol, College Stn, TX 77843 USA Texas A&M Univ, Dept Hlth & Kinesiol, College Stn, TX 77843 USA Publisher: HUMAN KINETICS PUBL INC, 1607 N MARKET ST, CHAMPAIGN, IL 61820- 2200 USA IDS Number: 009NJ ISSN: 0273-5024 CITED REFERENCES: 2003 J CITATION REPO : 2005 *I SCI INF CURR CONT 0620 : 2005 AMIN M PERSPECTIVES PUBLISH 1 : 1 2000 CHRISTENSON JA ACCREDITING KNOWLEDGE - JOURNAL STATURE AND CITATION IMPACT IN SOCIAL- SCIENCE SOCIAL SCIENCE QUARTERLY 66 : 964 1985 DILEVKO J Evaluating academic journals without impact factors for collection management decisions COLLEGE & RESEARCH LIBRARIES 63 : 562 2002 GARFIELD E CITATION ANALYSIS AS A TOOL IN JOURNAL EVALUATION - JOURNALS CAN BE RANKED BY FREQUENCY AND IMPACT OF CITATIONS FOR SCIENCE POLICY STUDIES SCIENCE 178 : 471 1972 HARTER SP ISI's impact factor as misnomer: A proposed new measure to assess journal impact JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 48 : 1146 1997 HOUSNER LD TEACHER COGNITION - DIFFERENCES IN PLANNING AND INTERACTIVE DECISION-MAKING BETWEEN EXPERIENCED AND INEXPERIENCED TEACHERS RESEARCH QUARTERLY FOR EXERCISE AND SPORT 56 : 45 1985 MCBRIDE RE Thoughtful decision making in physical education: A modest proposal QUEST 56 : 337 2004 MOED HF Towards appropriate indicators of journal impact SCIENTOMETRICS 46 : 575 1999 SIECK GC The "impact factor": what it means to the impact of applied physiology JOURNAL OF APPLIED PHYSIOLOGY 89 : 865 2000 From garfield at CODEX.CIS.UPENN.EDU Thu Mar 30 14:43:56 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Thu, 30 Mar 2006 14:43:56 -0500 Subject: Excerpt from : McBride R. "Impact factors for the Journal of Teaching in Physical Education - What are they and are they important? " JOURNAL OF TEACHING IN PHYSICAL EDUCATION 25 (1): 3-8 JAN 2006 Message-ID: Ron McBride: rmac at tamu.edu TITLE : Impact Factors for the Journal of Teaching in Physical Education - What are they and are they important? AUTHOR : Ron McBride SOURCE : Journal of Teaching in Physical Education 25(1). p.3-8, January 2006. Human Kinetics Publ. Inc. EXCERPT FROM ABOVE PAPER: Although the precise interpretation of impact factors has not been definitively assessed, Harter and Nisonger (1997) note that most researchers and scholars employ the term to state, or at least implicitly assume it to represent, a measure of journal quality. Christenson and Sigelman (1985) associate impact factors with a journal's scholarly influence. Rightly or wrongly, the higher a journal's impact factor, the greater the perception of quality and prestige is accorded to that journal by many members of the scholarly community. Should the impact factor of the Journal of Teaching in Physical Education (JTPE) be of concern to its constituents? As a current member of the Editorial Board and contributing author over the last 20 years, I must respond with an unequivocal "Yes!" I answer in the affirmative not because of an innate bias on my part, but because of Dilevko and Atkinson's (2002) observation that when "a journal is cited frequently, [it] indicates that scholars in the field deem it to be influential" (p.563). Ultimately, the status of measure of import attached to any journal comes from the community of scholars who acknowledge such status through cited articles. In this editorial article I address the importance of JTPE's impact factor from both a personal and profession perspective, offer support for my assessment, and provide suggestions for improving its impact factor. Before proceeding it is important to note that, admittedly, the impact factor represents just one indicator of assessed value, and a journal should not necessarily be judged solely on this one measure. Nevertheless, because impact factors remain one of the most accepted and utilized criteria for assessing journal quality in the scientific and educational communities, they represent the focal point for this commentary. Addresses: McBride R (reprint author), Texas A&M Univ, Dept Hlth & Kinesiol, College Stn, TX 77843 USA Texas A&M Univ, Dept Hlth & Kinesiol, College Stn, TX 77843 USA Publisher: HUMAN KINETICS PUBL INC, 1607 N MARKET ST, CHAMPAIGN, IL 61820- 2200 USA IDS Number: 009NJ ISSN: 0273-5024 CITED REFERENCES: 2003 J CITATION REPO : 2005 *I SCI INF CURR CONT 0620 : 2005 AMIN M PERSPECTIVES PUBLISH 1 : 1 2000 CHRISTENSON JA ACCREDITING KNOWLEDGE - JOURNAL STATURE AND CITATION IMPACT IN SOCIAL- SCIENCE SOCIAL SCIENCE QUARTERLY 66 : 964 1985 DILEVKO J Evaluating academic journals without impact factors for collection management decisions COLLEGE & RESEARCH LIBRARIES 63 : 562 2002 GARFIELD E CITATION ANALYSIS AS A TOOL IN JOURNAL EVALUATION - JOURNALS CAN BE RANKED BY FREQUENCY AND IMPACT OF CITATIONS FOR SCIENCE POLICY STUDIES SCIENCE 178 : 471 1972 HARTER SP ISI's impact factor as misnomer: A proposed new measure to assess journal impact JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 48 : 1146 1997 HOUSNER LD TEACHER COGNITION - DIFFERENCES IN PLANNING AND INTERACTIVE DECISION-MAKING BETWEEN EXPERIENCED AND INEXPERIENCED TEACHERS RESEARCH QUARTERLY FOR EXERCISE AND SPORT 56 : 45 1985 MCBRIDE RE Thoughtful decision making in physical education: A modest proposal QUEST 56 : 337 2004 MOED HF Towards appropriate indicators of journal impact SCIENTOMETRICS 46 : 575 1999 SIECK GC The "impact factor": what it means to the impact of applied physiology JOURNAL OF APPLIED PHYSIOLOGY 89 : 865 2000 From pmd8 at CORNELL.EDU Thu Mar 30 15:09:28 2006 From: pmd8 at CORNELL.EDU (Phil Davis) Date: Thu, 30 Mar 2006 15:09:28 -0500 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: Stephen, I wouldn't call you a "capitalist pig" but a willfully blind, cultural elitist. In countries where education is wholly (or mostly) funded by the government -- not just the UK and Europe, but Canada and others -- the government is concerned about making sure that everyone gets some modicum of funding. That does not mean a completely equitable rationing system, but it ensures a base-level of funding. In the United States, this base-level funding often comes from one's own department or college. Granted, the capitalist-approach you speak of does reward the best and greatest, and this Winner-takes-all approach does result in pioneering research, yet it only rewards the few. --Phil Davis Stephen Bensman wrote: >Speaking as a capitalist pig, the entire RAE system is just another example >of socialists hoisting themselves on their own petards. Point 1 below >contains the essence of the problem. The US has done pioneering work on >the evaluation of research-doctorate programs but was never silly enough to >allocate research resources on the basis of it. Luckily because these >evaluations were usually screwed up in some way. Allocation of research >resources was always done on a project-by-project basis by the NSF, NIH, >and others, with experts in the fields evaluating individual research >proposals. The Europeans have a tendency to overplan everything with >disastrous consequences--the disaster in Eastern Europe just being the >latest example of it. > >SB From notsjb at LSU.EDU Thu Mar 30 15:32:25 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Thu, 30 Mar 2006 14:32:25 -0600 Subject: Future UK RAEs to be Metrics-Based Message-ID: Gee, I consider myself anything but a cultural elitist. After all, I work at LSU. The basic problem of the RAE is that it is biased against an institution like LSU. At least under the American system, good researchers at a place like LSU have an even chance to obtain research funding, and many take advantage of this system. That way a good researcher maintains his independence and advance his career. This way LSU plays a major role as a launch pad for up and coming scientists. The British RAE always reminded me of the Tsarist system of krugovaia poruka, where all the peasants of a commune were held liable for communal taxes. This was the taxation system of serfdom, causing peasants to be chained to the commune, stifling individual initiative, thereby causing agricultural stagnation, and ultimately a violent revolution. If this makes me a cultural elitist, then so be it. SB Phil Davis @LISTSERV.UTK.EDU> on 03/30/2006 02:09:28 PM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based Stephen, I wouldn't call you a "capitalist pig" but a willfully blind, cultural elitist. In countries where education is wholly (or mostly) funded by the government -- not just the UK and Europe, but Canada and others -- the government is concerned about making sure that everyone gets some modicum of funding. That does not mean a completely equitable rationing system, but it ensures a base-level of funding. In the United States, this base-level funding often comes from one's own department or college. Granted, the capitalist-approach you speak of does reward the best and greatest, and this Winner-takes-all approach does result in pioneering research, yet it only rewards the few. --Phil Davis Stephen Bensman wrote: >Speaking as a capitalist pig, the entire RAE system is just another example >of socialists hoisting themselves on their own petards. Point 1 below >contains the essence of the problem. The US has done pioneering work on >the evaluation of research-doctorate programs but was never silly enough to >allocate research resources on the basis of it. Luckily because these >evaluations were usually screwed up in some way. Allocation of research >resources was always done on a project-by-project basis by the NSF, NIH, >and others, with experts in the fields evaluating individual research >proposals. The Europeans have a tendency to overplan everything with >disastrous consequences--the disaster in Eastern Europe just being the >latest example of it. > >SB From jonathan at LEVITT.NET Fri Mar 31 11:20:39 2006 From: jonathan at LEVITT.NET (Jonathan Levitt) Date: Fri, 31 Mar 2006 17:20:39 +0100 Subject: Future UK RAEs to be Metrics-Based Message-ID: Hi, Thanks Stephen and Phil for your interesting feedback. Although I understand Stephen's criticisms of the UK RAE, in my view, compared with the system that it replaced it has encouraged researchers and departments to focus more on research. This is not to say that the UK RAE is without problems, but hopefully some of these will be addressed when the current system is replaced. Best regards, Jonathan. ----- Original Message ----- From: "Stephen J Bensman" To: Sent: Thursday, March 30, 2006 9:32 PM Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Gee, I consider myself anything but a cultural elitist. After all, I work > at LSU. The basic problem of the RAE is that it is biased against an > institution like LSU. At least under the American system, good > researchers > at a place like LSU have an even chance to obtain research funding, and > many take advantage of this system. That way a good researcher maintains > his independence and advance his career. This way LSU plays a major role > as a launch pad for up and coming scientists. The British RAE always > reminded me of the Tsarist system of krugovaia poruka, where all the > peasants of a commune were held liable for communal taxes. This was the > taxation system of serfdom, causing peasants to be chained to the commune, > stifling individual initiative, thereby causing agricultural stagnation, > and ultimately a violent revolution. If this makes me a cultural elitist, > then so be it. > > SB > > > > > Phil Davis @LISTSERV.UTK.EDU> on 03/30/2006 02:09:28 PM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Stephen, I wouldn't call you a "capitalist pig" but a willfully blind, > cultural elitist. In countries where education is wholly (or mostly) > funded by the government -- not just the UK and Europe, but Canada and > others -- the government is concerned about making sure that everyone gets > some modicum of funding. That does not mean a completely equitable > rationing system, but it ensures a base-level of funding. In the United > States, this base-level funding often comes from one's own department or > college. Granted, the capitalist-approach you speak of does reward the > best and greatest, and this Winner-takes-all approach does result in > pioneering research, yet it only rewards the few. > > --Phil Davis > > > > Stephen Bensman wrote: > >>Speaking as a capitalist pig, the entire RAE system is just another > example >>of socialists hoisting themselves on their own petards. Point 1 below >>contains the essence of the problem. The US has done pioneering work on >>the evaluation of research-doctorate programs but was never silly enough > to >>allocate research resources on the basis of it. Luckily because these >>evaluations were usually screwed up in some way. Allocation of research >>resources was always done on a project-by-project basis by the NSF, NIH, >>and others, with experts in the fields evaluating individual research >>proposals. The Europeans have a tendency to overplan everything with >>disastrous consequences--the disaster in Eastern Europe just being the >>latest example of it. >> >>SB From notsjb at LSU.EDU Fri Mar 31 11:35:59 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Fri, 31 Mar 2006 10:35:59 -0600 Subject: Future UK RAEs to be Metrics-Based Message-ID: Jonathan, Thanks for the thanks. You just heard the view of a follower of the Austrian school of economics (Hayek, Schumpeter) that equates socialism with serfdom. And socialistic is what the RAE is. I am always suspicious of little authoritarians ginning up numbers with which they can dominate other people. Down here in Louisiana such people make us reach for our Rebel flags and Springfields. SB Jonathan Levitt @LISTSERV.UTK.EDU> on 03/31/2006 10:20:39 AM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based Hi, Thanks Stephen and Phil for your interesting feedback. Although I understand Stephen's criticisms of the UK RAE, in my view, compared with the system that it replaced it has encouraged researchers and departments to focus more on research. This is not to say that the UK RAE is without problems, but hopefully some of these will be addressed when the current system is replaced. Best regards, Jonathan. ----- Original Message ----- From: "Stephen J Bensman" To: Sent: Thursday, March 30, 2006 9:32 PM Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Gee, I consider myself anything but a cultural elitist. After all, I work > at LSU. The basic problem of the RAE is that it is biased against an > institution like LSU. At least under the American system, good > researchers > at a place like LSU have an even chance to obtain research funding, and > many take advantage of this system. That way a good researcher maintains > his independence and advance his career. This way LSU plays a major role > as a launch pad for up and coming scientists. The British RAE always > reminded me of the Tsarist system of krugovaia poruka, where all the > peasants of a commune were held liable for communal taxes. This was the > taxation system of serfdom, causing peasants to be chained to the commune, > stifling individual initiative, thereby causing agricultural stagnation, > and ultimately a violent revolution. If this makes me a cultural elitist, > then so be it. > > SB > > > > > Phil Davis @LISTSERV.UTK.EDU> on 03/30/2006 02:09:28 PM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Stephen, I wouldn't call you a "capitalist pig" but a willfully blind, > cultural elitist. In countries where education is wholly (or mostly) > funded by the government -- not just the UK and Europe, but Canada and > others -- the government is concerned about making sure that everyone gets > some modicum of funding. That does not mean a completely equitable > rationing system, but it ensures a base-level of funding. In the United > States, this base-level funding often comes from one's own department or > college. Granted, the capitalist-approach you speak of does reward the > best and greatest, and this Winner-takes-all approach does result in > pioneering research, yet it only rewards the few. > > --Phil Davis > > > > Stephen Bensman wrote: > >>Speaking as a capitalist pig, the entire RAE system is just another > example >>of socialists hoisting themselves on their own petards. Point 1 below >>contains the essence of the problem. The US has done pioneering work on >>the evaluation of research-doctorate programs but was never silly enough > to >>allocate research resources on the basis of it. Luckily because these >>evaluations were usually screwed up in some way. Allocation of research >>resources was always done on a project-by-project basis by the NSF, NIH, >>and others, with experts in the fields evaluating individual research >>proposals. The Europeans have a tendency to overplan everything with >>disastrous consequences--the disaster in Eastern Europe just being the >>latest example of it. >> >>SB From garfield at CODEX.CIS.UPENN.EDU Fri Mar 31 13:32:21 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Fri, 31 Mar 2006 13:32:21 -0500 Subject: Cohen, AM; Hersh, WR; Peterson, K; Yen, PY "Reducing workload in systematic review preparation using automated citation classification" Journal of the American Medical Informatics Association 13(2): 206-219, March-April 2006. Message-ID: Aaron Michael Cohen: E-mail: cohenaa at ohsu.edu William R. Hersh : E-mail: hersh at ohsu.edu FULL TEXT AVAILABLE AT : http://medir.ohsu.edu/~hersh/jamia-06-sysrev.pdf TITLE: Reducing workload in systematic review preparation using automated citation classification (Review, English) AUTHOR: Cohen, AM; Hersh, WR; Peterson, K; Yen, PY SOURCE: JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION 13 (2). MAR-APR 2006. p.206-219 ELSEVIER SCIENCE INC, NEW YORK ABSTRACT: Objective: To determine whether automated classification of document citations can be useful in reducing the time spent by experts reviewing journal articles for inclusion in updating systematic reviews of drug class efficacy for treatment of disease. Design: A test collection was built using the annotated reference files from 15 systematic drug class reviews. A voting perceptron-based automated citation classification system was constructed to classify each article as containing high-quality, drug class-specific evidence or not. Cross-validation experiments were performed to evaluate performance. Measurements: Precision, recall, and F-measure were evaluated at a range of sample weightings. Work saved over sampling at 95% recall was used as the measure of value to the review process. Results: A reduction in the number of articles needing manual review was found for 11 of the 15 drug review topics studied. For three of the topics, the reduction was 50% or greater. Conclusion: Automated document citation classification could be a useful tool in maintaining systematic reviews of the efficacy of drug therapy. Further work is needed to refine the classification system and determine the best manner to integrate the system into the production of systematic reviews. AUTHOR ADDRESS: AM Cohen, Oregon Hlth & Sci Univ, Sch Med, Dept Med Informat & Clin Epidemiol, 3181 SW Sam Jackson Pk Rd,Mail Code BICC, Portland, OR 97239 USA From garfield at CODEX.CIS.UPENN.EDU Fri Mar 31 16:03:39 2006 From: garfield at CODEX.CIS.UPENN.EDU (Eugene Garfield) Date: Fri, 31 Mar 2006 16:03:39 -0500 Subject: Banville DL. "Mining chemical structural information from the drug literature (Review)" Drug Discovery Today, Volume 11, Issues 1-2, January 2006, P.35-42. Message-ID: AUTHOR : Debra L. Banville, debra.banville at astrazeneca.com TITLE : Mining chemical structural information from the drug literature (Review) SOURCE: Drug Discovery Today, Volume 11, Issues 1-2, January 2006, P.35-42. ADDRESS: AstraZeneca Pharmaceuticals, 1800 Concord Pike, Wilmington, DE 19850, USA Full text of this article is available. Copy and paste following url in the address window of your browser http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6T64-4J853YK- 6&_user=528388&_handle=V-WA-A-W-AB-MsSAYZW-UUA-U-AAVCAZZWZZ-AAVWDVDUZZ- DBAYEDEVE-AB-U&_fmt=summary&_coverDate=01%2F31% 2F2006&_rdoc=6&_orig=browse&_srch=%23toc%235020%232006%23999889998%23616646! &_cdi=5020&view=c&_acct=C000005838&_version=1&_urlVersion=0&_userid=528388&m d5=20c64a72dec6603830ae6953bd54b88e ABSTRACT: It is easier to find too many documents on a life science topic than to find the right information inside these documents. With the application of text data mining to biological documents, it is no surprise that researchers are starting to look at applications that mine out chemical information. The mining of chemical entities ? names and structures ? brings with it some unique challenges, which commercial and academic efforts are beginning to address. Ultimately, life science text data mining applications need to focus on the marriage of biological and chemical information. From loet at LEYDESDORFF.NET Fri Mar 31 12:57:46 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Fri, 31 Mar 2006 19:57:46 +0200 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: Dear Stephen, Although I am politically at the other end of the spectrum, I fully agree with your critique of the RAE. But the critique would equally hold for a "metric" that would rate departments against each other as proposed by some of our colleagues. The problem is to take departments as units of analysis. With best wishes, Loet > -----Original Message----- > From: ASIS&T Special Interest Group on Metrics > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Stephen J Bensman > Sent: Thursday, March 30, 2006 10:32 PM > To: SIGMETRICS at listserv.utk.edu > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Gee, I consider myself anything but a cultural elitist. > After all, I work at LSU. The basic problem of the RAE is > that it is biased against an institution like LSU. At least > under the American system, good researchers at a place like > LSU have an even chance to obtain research funding, and many > take advantage of this system. That way a good researcher > maintains his independence and advance his career. This way > LSU plays a major role as a launch pad for up and coming > scientists. The British RAE always reminded me of the > Tsarist system of krugovaia poruka, where all the peasants of > a commune were held liable for communal taxes. This was the > taxation system of serfdom, causing peasants to be chained to > the commune, stifling individual initiative, thereby causing > agricultural stagnation, and ultimately a violent revolution. > If this makes me a cultural elitist, then so be it. > > SB > > > > > Phil Davis @LISTSERV.UTK.EDU> on 03/30/2006 > 02:09:28 PM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Stephen, I wouldn't call you a "capitalist pig" but a > willfully blind, cultural elitist. In countries where > education is wholly (or mostly) funded by the government -- > not just the UK and Europe, but Canada and others -- the > government is concerned about making sure that everyone gets > some modicum of funding. That does not mean a completely > equitable rationing system, but it ensures a base-level of > funding. In the United States, this base-level funding often > comes from one's own department or college. Granted, the > capitalist-approach you speak of does reward the best and > greatest, and this Winner-takes-all approach does result in > pioneering research, yet it only rewards the few. > > --Phil Davis > > > > Stephen Bensman wrote: > > >Speaking as a capitalist pig, the entire RAE system is just another > example > >of socialists hoisting themselves on their own petards. > Point 1 below > >contains the essence of the problem. The US has done > pioneering work > >on the evaluation of research-doctorate programs but was never silly > >enough > to > >allocate research resources on the basis of it. Luckily > because these > >evaluations were usually screwed up in some way. Allocation of > >research resources was always done on a project-by-project > basis by the > >NSF, NIH, and others, with experts in the fields evaluating > individual > >research proposals. The Europeans have a tendency to overplan > >everything with disastrous consequences--the disaster in > Eastern Europe > >just being the latest example of it. > > > >SB >