[Sigmetrics] "Use of the journal impact factor for assessing individual articles need not be wrong" by Ludo Waltman and Vincent Traag

Fri Mar 10 13:36:25 EST 2017

IF is a good indicator of journal quality, because of the place of the review article in the structure of scientific communication.  It is on this basis that Garfield made it his main indicator of journal  quality.   Review journals have the highest IF, and this is because of their place in the social structure of science.  You have to be picked to write a review article, and you are picked because editors know you.  I discuss all this in a number of my articles Garfield has posted on his Web site at the following URL:

http://www.garfield.library.upenn.edu/bensman/bensman.html

Stephen J. Bensman - Papers on Urquhart's Law and Garfield ...<http://www.garfield.library.upenn.edu/bensman/bensman.html>
www.garfield.library.upenn.edu
Stephen J. Bensman - Bensman SJ "Eugene Garfield, Francis Narin, and Pagerank: The theoretical bases of the Google Search Engine" Archives, Cornell University, 2014.

I spent my entire career studying Eugene Garfield and why he is important.  Basically I am a historian, and historians make other people famous,  not themselves.  You can go to my Google Scholar citations page and get further insights into these matters.  I only hope that people will find that I have done him justice.  He also did lay the theoretical bases of the Google search engines by showing that relationships are semantically correct and not words.

Respectfully,

Stephen J. Bensman, Ph.D.

LSU Libraries (Retired)

________________________________
From: SIGMETRICS <sigmetrics-bounces at asis.org> on behalf of David Wojick <dwojick at craigellachie.us>
Sent: Friday, March 10, 2017 11:49 AM
To: SIGMetrics
Subject: Re: [Sigmetrics] "Use of the journal impact factor for assessing individual articles need not be wrong" by Ludo Waltman and Vincent Traag

I have a rather different conjecture as to why the IF is a useful proxy for article quality. It has to do with the tiered structure of the system of journals. Journals with high IFs tend to attract many submissions and therefore have high rejection rates. Thus there is extensive article level evaluation. Only those articles judged to be of the highest quality get published. It follows that getting published in a high IF journal is indicative of relatively high quality for the article.

Opponents of IF argue that articles should be individually evaluated at the institutional level, but the high IF journal is better positioned to do this, because it has a large sample from many authors.

David

David Wojick, Ph.D.
http://insidepublicaccess.com/

At 07:17 AM 3/10/2017, Loet Leydesdorff wrote:

Inside Public Access<http://insidepublicaccess.com/>
insidepublicaccess.com
Welcome to Inside Public Access, where experts and insiders track the US Public Access program. The US Government has embarked on a massive new public access program ...

re: "Use of the journal impact factor for assessing individual articles need not be wrong" by Ludo Waltman and Vincent Traag at https://www.cwts.nl/blog…<https://l.facebook.com/l.php?u=https%3A%2F%2Fwww.cwts.nl%2Fblog%3Farticle%3Dn-q2z254%26title%3Duse-of-the-journal-impact-factor-for-assessing-individual-articles-need-not-be-wrong&h=ATMOIsCG9ATnRfQ-OlFJkFeiHkoCcRivWRZfeAzZmaLvdRhfQRDY1bd__tb3hFnW9DnDno9cQBVZacIbGHbAPnKz88t0IcumxxRBBRjEFwHkT--PrzdlCAzu7hYvKNiDAD7TFedL8C-pPQ&enc=AZPwLn5krg8kNa-HStWph2-10TOjcwx_yqfRy9azLwS063lMs9eEsHFxloKm6BEGWaym7RN5cSAc9QeYLUfXiL_M3Zo1RPQyXD1HZKDgacgKfy_QybLl9NI5PG71e2vSQ1cyQtxbx7Q6RkHeEsAhapHhGAQmmzDI2oTsTXlwHTvTSfPNszAxAcoiVy29sCMNsGc&s=1>

I agree with the authors that there are two different arguments against using the impact factor of a journal (IF) as a proxy for the quality of papers in the journal: (1) the skewness of the citation distribution, and (2) the ecological fallacy.

1. Against argument 1, the authors reason as follows: Let us assume (in scenario 2, at p. 16) that “journals are relatively homogenous in terms of the values of the articles they publish.” This relatively flat distribution of the non-observable “values” is for (unknown) statistical reasons represented by the skewed distribution of citations to these articles. The latter distribution can be observed. In this scenario, a journal measure such as the journal impact factorin other words, the meancould be a better predictor of the “value” of an article than its individual citation rate.

Unlike the reasoning of others who criticize the use of the IF for the evaluation of individual papers, the reasoning above would be free of assumptions. [https://www.facebook.com/images/emoji.php/v7/feb/2/16/1f642.png]  :-)

2. Let me add that the ecological fallacy (Robinson, 1950) does not imply that the value of an attribute to an individual is independent of the value at the group level, but that the latter may fail as a predictor of the former. One loses control of the prediction: in some cases it works; in others not.

See: Kreft, G. G., & de Leeuw, E. (1988). The see-saw effect: A multilevel problem? Quality and Quantity, 22(2), 127-137.
Abstract: Studies of school effectiveness often use measures of association, such as regression weights and correlation coefficients. These statistics are used to estimate the size of the change or “effect” that would occur in one variable (for example reading ability) given a particular change in another variable (for example sex and sex ratio). In this paper we explore the limitations of regression coefficients for use in a contextual analysis, in which both individual and contextual variables are included as independent variables. In our example “individual sex” and a context variable “sex ratio of the schoolclass” are regressors, and reading ability is the dependent variable. Our conclusion is that researchers should be careful making interpretations of effects from multiple regression analysis, when dealing with aggregate data. Even in the case (as in our example) when individual and contextual variables are made orthogonal to avoid multicollinearity, interpretation of the effects of the aggregate variable is problematical.