[Sigmetrics] New paper

[William Gunn]

Hi Lutz,

I've read your paper with interest & I think the analysis is well done,
though I have to say pre-registration of your study would have strengthened
the findings, given the small effect sizes you report.

I had a few questions & would be grateful for any response:

The main question I had was if you plan to do any follow-up work to
disentangle the correlation between presence at an elite institution,
publication in a high IF journal, and higher mean or total normalized
citations. It seems to me, not being as familiar with the trends among
indicators as you, that you have provided nearly equal support for two
different ways of picking early investigators likely to be productive:
picking them according to Q1 as you describe or picking the ones which are
at elite institutions early in their career (as well as picking according
to number of papers). Just wondering if you're planning to try to get at
causality in some way among these interrelated factors?

Other things that occurred to me during reading:
Why do you think profiles manually created by researchers will be better
than profiles automatically generated and then edited?
Instead of using publication early in the career and publication late in
career to define a cohort which presumably published continuously, couldn't
you write a query, since you have the data, to actually select only those
who have indeed published continuously?
Am I correct that the main difference between the three figures is that
there's a smaller time window in 2 than 1 and 3 than 2?
Could you explain the reversion in mean citations of the upper cohorts over
time in terms of the divided attention allocated to the increased overall
publication output? In other words, could it be that as the overall number
of publications grows, attention gets further divided and mean citation
rates fall?
Would you expect to see the same results using CiteScore?

Again, grateful for any response!


William Gunn
+1 (650) 614-1749
http://synthesis.williamgunn.org/about/

On Wed, Jun 21, 2017 at 5:26 AM, Bornmann, Lutz <lutz.bornmann at gv.mpg.de>
wrote:

> Dear colleague,
>
>
>
> You might be interested in the following paper:
>
>
>
> Can the Journal Impact Factor Be Used as a Criterion for the Selection of
> Junior Researchers? A Large-Scale Empirical Study Based on ResearcherID Data
>
>
>
> Early in researchers' careers, it is difficult to assess how good their
> work is or how important or influential the scholars will eventually be.
> Hence, funding agencies, academic departments, and others often use the
> Journal Impact Factor (JIF) of where the authors have published to assess
> their work and provide resources and rewards for future work. The use of
> JIFs in this way has been heavily criticized, however. Using a large data
> set with many thousands of publication profiles of individual researchers,
> this study tests the ability of the JIF (in its normalized variant) to
> identify, at the beginning of their careers, those candidates who will be
> successful in the long run. Instead of bare JIFs and citation counts, the
> metrics used here are standardized according to Web of Science subject
> categories and publication years. The results of the study indicate that
> the JIF (in its normalized variant) is able to discriminate between
> researchers who published papers later on with a citation impact above or
> below average in a field and publication year - not only in the short term,
> but also in the long term. However, the low to medium effect sizes of the
> results also indicate that the JIF (in its normalized variant) should not
> be used as the sole criterion for identifying later success: other
> criteria, such as the novelty and significance of the specific research,
> academic distinctions, and the reputation of previous institutions, should
> also be considered.
>
>
>
> Available at: https://arxiv.org/abs/1706.06515
>
>
>
> Best,
>
>
>
> Lutz
>
>
>
> ---------------------------------------
>
>
>
> Dr. Dr. habil. Lutz Bornmann
>
> Division for Science and Innovation Studies
>
> Administrative Headquarters of the Max Planck Society
>
> Hofgartenstr. 8
>
> 80539 Munich
>
> Tel.: +49 89 2108 1265 <+49%2089%2021081265>
>
> Mobil: +49 170 9183667 <+49%20170%209183667>
>
> Email: bornmann at gv.mpg.de
>
> WWW: www.lutz-bornmann.de
>
> ResearcherID: http://www.researcherid.com/rid/A-3926-2008
>
> ResearchGate: http://www.researchgate.net/profile/Lutz_Bornmann
>
>
>
> _______________________________________________
> SIGMETRICS mailing list
> SIGMETRICS at mail.asis.org
> http://mail.asis.org/mailman/listinfo/sigmetrics
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20170621/355bdc31/attachment-0001.html>