Crowd-Sourced Peer Review: Substitute or Supplement?

Pikas, Christina K. Christina.Pikas at JHUAPL.EDU
Mon Aug 25 07:06:26 EDT 2014

The part that stood out for me is: "affiliated with an accredited university" . I guess all the industry scientists in petroleum, pharmaceuticals, biotechnology, materials, aerospace... etc., are ineligible? Lovely.

From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at] On Behalf Of Stevan Harnad
Sent: Friday, August 22, 2014 12:02 PM
Subject: Re: [SIGMETRICS] Crowd-Sourced Peer Review: Substitute or Supplement?

Only Scientific Members who are affiliated with an accredited university and have at least
5 publications assigned to their ORCID account may officially review an article at ScienceOpen.

So  being affiliated with an accredited university and having at least 5 publications makes
anyone a specialist qualified to review anyone's research?

The criterion for peership might have to be a trifle more exacting than that even before we
raise once again the niggling question of answerability...

Harnad, S. (1990) Scholarly Skywriting and the Prepublication Continuum of Scientific Inquiry
Psychological Science 1: 342 - 343 (reprinted in Current Contents 45: 9-13, November 11 1991).

Stevan Harnad

On Aug 22, 2014, at 4:41 AM, Jennifer Smith <jesmith at<mailto:jesmith at>> wrote:

Hi All

As list members probably know, Frontiers In ... have a slightly different method of reviewing<> than traditional peer review model.

I noticed recently another alternative model for reviewing, at ScienceOpen (based on having an academic ID, in this case, ORCiD):

"Comments and Reviews require registration via ORCID

Everybody can read, download and share your article. Commenting, rating and reviewing,
however, requires previous registration via ORCID. Only Scientific Members who are
affiliated with an accredited university and have at least 5 publications assigned to their
ORCID account may officially review an article at ScienceOpen. Commenting requires
at least 1 publication. Please refer to our User Categories for a detailed description. In
any case, please consult our Peer Review Guidelines and Guidelines for Commenting
before writing a review or commenting on papers."

Interesting times to see what develops and is taken up.

Kind regards,


Jennifer Smith
Research Publications Librarian
Information Services
St George's University of London
E: jesmith at<mailto:jesmith at>
T: +44 (0)20 8725 5393

From: Repositories discussion list [mailto:JISC-REPOSITORIES at JISCMAIL.AC.UK] On Behalf Of Stevan Harnad
Sent: 21 August 2014 20:19
Subject: Crowd-Sourced Peer Review: Substitute or Supplement?

Harnad, S. (2014) Crowd-Sourced Peer Review: Substitute or supplement for the current outdated system?<> LSE Impact Blog 8/21


If, as rumoured<>, google builds a platform for depositing unrefereed research papers for "peer-reviewing" viacrowd-sourcing<>, can this create a substitute<> for classical peer-review or will it merely supplement<> classical peer review with crowd-sourcing?

... no one knows whether crowd-sourced peer-review, even if it could work, would be scaleable or sustainable.

The key questions are hence:
1. Would all (most? many?) authors be willing to post their unrefereed papers publicly (and in place of submitting them to journals!)?

2. Would all (most? many?) of the posted papers attract referees? competent experts?

3. Who/what decides whether the refereeing is competent, and whether the author has adequately complied? (Relying on a Wikipedia-style cadre of 2nd-order crowd-sourcers<> who gain authority recursively in proportion to how much 1st-order crowd-sourcing they have done - rather than on the basis of expertise -  sounds like a way to generate Wikipedia quality, but not peer-reviewed quality...)

4. If any of this actually happens on any scale, will it be sustainable?

5. Would this make the landscape (unrefereed preprints, referee comments, revised postprints) as navigable and useful as classical peer review, or not?
My own prediction (based on nearly a quarter century of umpiring<> both classical peer review and open peer commentary) is that crowdsourcing will provide an excellent supplement to classical peer review but not a substitute for it. Radical implementations will simply end up re-inventing classical peer review, but on a much faster and more efficient PostGutenberg platform. We will not realize this, however, until all of the peer-reviewed literature has first been made open access. And for that it is not sufficient for Google merely to provide a platform for authors to put their unrefereed papers, because most authors don't even put their refereed papers in their institutional repositories until it is mandated by their institutions and funders.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the SIGMETRICS mailing list