New Papers

[Smalheiser, Neil]

I think the most important question here is: Is it fair that one group of
reviewers rate manuscripts more favourably than another group - independently of the quality of the submitted manuscripts?

Dear Lutz,

In a court of law, that would be called a "leading question", because it assumes the answer. Who decides the quality of the submitted manuscripts? The reviewers, at least in this context (though ultimately it is the readers and the scientific community who decide). If there were an objective measure of quality, we could do without peer review entirely.

This also assumes that authors tend to recommend their personal friends who have conflicts of interest and who are less qualified to give objective opinions than the ones chosen by editors. In my own experience as an editor, authors tend to recommend leaders in their field as potential reviewers, often the same people I would have chosen as reviewers; and the minority of authors who recommend outliers are quickly flagged and over-ridden.

Moreover, there is also the hidden assumption that authors act to maximize the chances that their manuscripts will be accepted. I have to tell you that my latest research paper (on endogenous siRNAs in brain) is on a controversial subject, so I deliberately sent it to the most mainstream journal in the field where it would be scrutinized by the most skeptical molecular biologists -- rather than sending it to some neuroscience journal where it might be accepted more readily. Why? Because I know that molecular biologists won't believe something they read in a neuroscience journal; unless they have blessed it themselves, it does not exist. I did submit a list of 5 potential reviewers, who I knew were familiar with neuroscience -- whereas most of the journal's reviewers deal with yeast or C. elegans. This probably did help my paper win acceptance, because reviewers who are familiar with the specific topic are more likely to understand the novelty and innovation better than others.

In terms of fairness, the answer to your question is a strong YES. If I submit a manuscript, and it has errors or embarrassing flaws, I would expect my friends to be especially alert and protective on my behalf! Conversely, I can give you many examples where leading journals have relied on certain prominent reviewers who have prevented publication of innovative articles and thereby have ruined careers [one cannot be funded if one cannot publish the findings] and slowed down entire research fields. These reviewers think that they are being objective and simply insisting on quality. The author-suggested reviewers is one of the few ways around this roadblock. Indeed, historically, one of the main reasons that new journals arise is because a certain type or line of research is not taken seriously by the established journals. Just this year, the ACM (computer science assn.) has established their own intl. conference and journal, because the existing society (AMIA) does not recognize or appreciate computer science-oriented submissions in their own conference or journal. I don't think AMIA has any higher quality standards than ACM. AMIA reviewers simply do not resonate with the types of questions and approaches that CS authors have -- yet I bet THEY feel that they are acting as quality gatekeepers.

I think there is a larger question here, which is whether one can usefully analyze scientific behavior strictly from the outside, in a black-box input-output manner, without modeling the internal machinery. YES, I do support that perspective (and if my colleague Vetle Torvik is listening, this is similar to the data-mining analysis of collaboration networks that he is initiating). However, it is only one facet of an overall analysis that also has to understand the behavior of scientists in terms of their own stated practice and reasons.

Neil Smalheiser