[Sigmetrics] "Use of the journal impact factor for assessing individual articles need not be wrong" by Ludo Waltman and Vincent Traag

[David Wojick]

I have a rather different conjecture as to why the IF is a useful proxy for 
article quality. It has to do with the tiered structure of the system of 
journals. Journals with high IFs tend to attract many submissions and 
therefore have high rejection rates. Thus there is extensive article level 
evaluation. Only those articles judged to be of the highest quality get 
published. It follows that getting published in a high IF journal is 
indicative of relatively high quality for the article.

Opponents of IF argue that articles should be individually evaluated at the 
institutional level, but the high IF journal is better positioned to do 
this, because it has a large sample from many authors.

David

David Wojick, Ph.D.
http://insidepublicaccess.com/

At 07:17 AM 3/10/2017, Loet Leydesdorff wrote:

>re: "Use of the journal impact factor for assessing individual articles 
>need not be wrong" by Ludo Waltman and Vincent Traag at 
><https://l.facebook.com/l.php?u=https%3A%2F%2Fwww.cwts.nl%2Fblog%3Farticle%3Dn-q2z254%26title%3Duse-of-the-journal-impact-factor-for-assessing-individual-articles-need-not-be-wrong&h=ATMOIsCG9ATnRfQ-OlFJkFeiHkoCcRivWRZfeAzZmaLvdRhfQRDY1bd__tb3hFnW9DnDno9cQBVZacIbGHbAPnKz88t0IcumxxRBBRjEFwHkT--PrzdlCAzu7hYvKNiDAD7TFedL8C-pPQ&enc=AZPwLn5krg8kNa-HStWph2-10TOjcwx_yqfRy9azLwS063lMs9eEsHFxloKm6BEGWaym7RN5cSAc9QeYLUfXiL_M3Zo1RPQyXD1HZKDgacgKfy_QybLl9NI5PG71e2vSQ1cyQtxbx7Q6RkHeEsAhapHhGAQmmzDI2oTsTXlwHTvTSfPNszAxAcoiVy29sCMNsGc&s=1>https://www.cwts.nl/blog
 
>
>
>I agree with the authors that there are two different arguments against 
>using the impact factor of a journal (IF) as a proxy for the quality of 
>papers in the journal: (1) the skewness of the citation distribution, and 
>(2) the ecological fallacy.
>
>1. Against argument 1, the authors reason as follows: Let us assume (in 
>scenario 2, at p. 16) that “journals are relatively homogenous in terms of 
>the values of the articles they publish.” This relatively flat 
>distribution of the non-observable “values” is for (unknown) statistical 
>reasons represented by the skewed distribution of citations to these 
>articles. The latter distribution can be observed. In this scenario, a 
>journal measure such as the journal impact factor­in other words, the 
>mean­could be a better predictor of the “value” of an article than its 
>individual citation rate.
>
>Unlike the reasoning of others who criticize the use of the IF for the 
>evaluation of individual papers, the reasoning above would be free of 
>assumptions.
>https://www.facebook.com/images/emoji.php/v7/feb/2/16/1f642.png
>:-)
>
>2. Let me add that the ecological fallacy (Robinson, 1950) does not imply 
>that the value of an attribute to an individual is independent of the 
>value at the group level, but that the latter may fail as a predictor of 
>the former. One loses control of the prediction: in some cases it works; 
>in others not.
>
>See: Kreft, G. G., & de Leeuw, E. (1988). The see-saw effect: A multilevel 
>problem? Quality and Quantity, 22(2), 127-137.
>Abstract: Studies of school effectiveness often use measures of 
>association, such as regression weights and correlation coefficients. 
>These statistics are used to estimate the size of the change or “effect” 
>that would occur in one variable (for example reading ability) given a 
>particular change in another variable (for example sex and sex ratio). In 
>this paper we explore the limitations of regression coefficients for use 
>in a contextual analysis, in which both individual and contextual 
>variables are included as independent variables. In our example 
>“individual sex” and a context variable “sex ratio of the schoolclass” are 
>regressors, and reading ability is the dependent variable. Our conclusion 
>is that researchers should be careful making interpretations of effects 
>from multiple regression analysis, when dealing with aggregate data. Even 
>in the case (as in our example) when individual and contextual variables 
>are made orthogonal to avoid multicollinearity, interpretation of the 
>effects of the aggregate variable is problematical.
>
>See also:
>
>·       Robinson, W. D. (1950). Ecological correlations and the behavior 
>of individuals. American Sociological Review, 15, 351-357.
>
>·       Leydesdorff, L., Wouters, P., & Bornmann, L. (2016). Professional 
>and citizen bibliometrics: complementarities and ambivalences in the 
>development and use of indicators­a state-of-the-art report. 
>Scientometrics, 109(3), 2129-2150.
>
>Best wishes,
>Loet
>
>
>Loet Leydesdorff
>Professor, University of Amsterdam
>Amsterdam School of Communication Research (ASCoR)
><mailto:loet at leydesdorff.net>loet at leydesdorff.net ; 
>http://www.leydesdorff.net/
>Associate Faculty, <http://www.sussex.ac.uk/spru/>SPRU, University of Sussex;
>Guest Professor <http://www.zju.edu.cn/english/>Zhejiang Univ., Hangzhou; 
>Visiting Professor, <http://www.istic.ac.cn/Eng/brief_en.html>ISTIC, Beijing;
>Visiting Fellow, <http://www.bbk.ac.uk/>Birkbeck, University of London;
><http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en>http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en
>
>
>
>_______________________________________________
>SIGMETRICS mailing list
>SIGMETRICS at mail.asis.org
>http://mail.asis.org/mailman/listinfo/sigmetrics
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20170310/3e15ed84/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 127d7be.png
Type: image/png
Size: 1447 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20170310/3e15ed84/attachment.png>