[Sigmetrics] "Use of the journal impact factor for assessing individual articles need not be wrong" by Ludo Waltman and Vincent Traag
dwojick at craigellachie.us
Fri Mar 10 12:49:30 EST 2017
I have a rather different conjecture as to why the IF is a useful proxy for
article quality. It has to do with the tiered structure of the system of
journals. Journals with high IFs tend to attract many submissions and
therefore have high rejection rates. Thus there is extensive article level
evaluation. Only those articles judged to be of the highest quality get
published. It follows that getting published in a high IF journal is
indicative of relatively high quality for the article.
Opponents of IF argue that articles should be individually evaluated at the
institutional level, but the high IF journal is better positioned to do
this, because it has a large sample from many authors.
David Wojick, Ph.D.
At 07:17 AM 3/10/2017, Loet Leydesdorff wrote:
>re: "Use of the journal impact factor for assessing individual articles
>need not be wrong" by Ludo Waltman and Vincent Traag at
>I agree with the authors that there are two different arguments against
>using the impact factor of a journal (IF) as a proxy for the quality of
>papers in the journal: (1) the skewness of the citation distribution, and
>(2) the ecological fallacy.
>1. Against argument 1, the authors reason as follows: Let us assume (in
>scenario 2, at p. 16) that journals are relatively homogenous in terms of
>the values of the articles they publish. This relatively flat
>distribution of the non-observable values is for (unknown) statistical
>reasons represented by the skewed distribution of citations to these
>articles. The latter distribution can be observed. In this scenario, a
>journal measure such as the journal impact factorin other words, the
>meancould be a better predictor of the value of an article than its
>individual citation rate.
>Unlike the reasoning of others who criticize the use of the IF for the
>evaluation of individual papers, the reasoning above would be free of
>2. Let me add that the ecological fallacy (Robinson, 1950) does not imply
>that the value of an attribute to an individual is independent of the
>value at the group level, but that the latter may fail as a predictor of
>the former. One loses control of the prediction: in some cases it works;
>in others not.
>See: Kreft, G. G., & de Leeuw, E. (1988). The see-saw effect: A multilevel
>problem? Quality and Quantity, 22(2), 127-137.
>Abstract: Studies of school effectiveness often use measures of
>association, such as regression weights and correlation coefficients.
>These statistics are used to estimate the size of the change or effect
>that would occur in one variable (for example reading ability) given a
>particular change in another variable (for example sex and sex ratio). In
>this paper we explore the limitations of regression coefficients for use
>in a contextual analysis, in which both individual and contextual
>variables are included as independent variables. In our example
>individual sex and a context variable sex ratio of the schoolclass are
>regressors, and reading ability is the dependent variable. Our conclusion
>is that researchers should be careful making interpretations of effects
>from multiple regression analysis, when dealing with aggregate data. Even
>in the case (as in our example) when individual and contextual variables
>are made orthogonal to avoid multicollinearity, interpretation of the
>effects of the aggregate variable is problematical.
>· Robinson, W. D. (1950). Ecological correlations and the behavior
>of individuals. American Sociological Review, 15, 351-357.
>· Leydesdorff, L., Wouters, P., & Bornmann, L. (2016). Professional
>and citizen bibliometrics: complementarities and ambivalences in the
>development and use of indicatorsa state-of-the-art report.
>Scientometrics, 109(3), 2129-2150.
>Professor, University of Amsterdam
>Amsterdam School of Communication Research (ASCoR)
><mailto:loet at leydesdorff.net>loet at leydesdorff.net ;
>Associate Faculty, <http://www.sussex.ac.uk/spru/>SPRU, University of Sussex;
>Guest Professor <http://www.zju.edu.cn/english/>Zhejiang Univ., Hangzhou;
>Visiting Professor, <http://www.istic.ac.cn/Eng/brief_en.html>ISTIC, Beijing;
>Visiting Fellow, <http://www.bbk.ac.uk/>Birkbeck, University of London;
>SIGMETRICS mailing list
>SIGMETRICS at mail.asis.org
-------------- next part --------------
An HTML attachment was scrubbed...
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 1447 bytes
Desc: not available
More information about the SIGMETRICS