Crowd-Sourced Peer Review: Substitute or Supplement?

Mon Aug 25 12:34:21 EDT 2014

Not to mention scientists working for government agencies, laboratories, associations, foundations ... or even publishers. As I recall, editors of learned periodicals have the power to accept or reject without further consultation.

Albert Henderson, former editor
Publishing Research Quarterly 

-----Original Message-----
From: Pikas, Christina K. <Christina.Pikas at JHUAPL.EDU>
To: SIGMETRICS <SIGMETRICS at LISTSERV.UTK.EDU>
Sent: Mon, Aug 25, 2014 7:06 am
Subject: Re: [SIGMETRICS] Crowd-Sourced Peer Review: Substitute or Supplement?

Adminstrative info for SIGMETRICS (for example unsubscribe):http://web.utk.edu/~gwhitney/sigmetrics.html

The part that stood out for me is: “affiliated with an accredited university” . I guess all the industry scientists in petroleum, pharmaceuticals, biotechnology, materials, aerospace… etc., are ineligible? Lovely.

From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at listserv.utk.edu]On Behalf Of Stevan Harnad
Sent: Friday, August 22, 2014 12:02 PM
To: SIGMETRICS at listserv.utk.edu
Subject: Re: [SIGMETRICS] Crowd-Sourced Peer Review: Substitute or Supplement?

Adminstrative info for SIGMETRICS (for example unsubscribe):http://web.utk.edu/~gwhitney/sigmetrics.html
Only Scientific Members who are affiliated with an accredited university and have at least

5 publications assigned to their ORCID account may officially review an article at ScienceOpen. 

So  being affiliated with an accredited university and having at least 5 publications makes 

anyone a specialist qualified to review anyone's research?

The criterion for peership might have to be a trifle more exacting than that even before we

raise once again the niggling question of answerability...

Harnad, S. (1990) Scholarly Skywriting and the Prepublication Continuum of Scientific Inquiry 

Psychological Science 1: 342 - 343 (reprinted in Current Contents 45: 9-13, November 11 1991). 

http://cogprints.org/1581/

Stevan Harnad

On Aug 22, 2014, at 4:41 AM, Jennifer Smith <jesmith at sgul.ac.uk> wrote:

Hi All

As list members probably know, Frontiers In … have a slightly different method of reviewing than traditional peer review model.

I noticed recently another alternative model for reviewing, at ScienceOpen (based on having an academic ID, in this case, ORCiD):

“Comments and Reviews require registration via ORCID

Everybody can read, download and share your article. Commenting, rating and reviewing,

however, requires previous registration via ORCID. Only Scientific Members who are

affiliated with an accredited university and have at least 5 publications assigned to their

ORCID account may officially review an article at ScienceOpen. Commenting requires

at least 1 publication. Please refer to our User Categories for a detailed description. In

any case, please consult our Peer Review Guidelines and Guidelines for Commenting

before writing a review or commenting on papers.”

http://about.scienceopen.com/how-does-it-work/#more-9

Interesting times to see what develops and is taken up.

Kind regards,

Jennifer

Jennifer Smith

Research Publications Librarian

Library

Information Services

St George's University of London

E: jesmith at sgul.ac.uk

T: +44 (0)20 8725 5393

From: Repositories discussion list [mailto:JISC-REPOSITORIES at JISCMAIL.AC.UK] On Behalf Of Stevan Harnad
Sent: 21 August 2014 20:19
To: JISC-REPOSITORIES at JISCMAIL.AC.UK
Subject: Crowd-Sourced Peer Review: Substitute or Supplement?

Harnad, S. (2014) Crowd-Sourced Peer Review: Substitute or supplement for the current outdated system? LSE Impact Blog 8/21 

EXCERPT:

If, as rumoured, google builds a platform for depositing unrefereed research papers for “peer-reviewing” viacrowd-sourcing, can this create a substitute for classical peer-review or will it merely supplement classical peer review with crowd-sourcing?

... no one knows whether crowd-sourced peer-review, even if it could work, would be scaleable or sustainable.

The key questions are hence:

1. Would all (most? many?) authors be willing to post their unrefereed papers publicly (and in place of submitting them to journals!)?

2. Would all (most? many?) of the posted papers attract referees? competent experts?

3. Who/what decides whether the refereeing is competent, and whether the author has adequately complied? (Relying on a Wikipedia-style cadre of 2nd-order crowd-sourcers who gain authority recursively in proportion to how much 1st-order crowd-sourcing they have done — rather than on the basis of expertise —  sounds like a way to generate Wikipedia quality, but not peer-reviewed quality…)

4. If any of this actually happens on any scale, will it be sustainable?

5. Would this make the landscape (unrefereed preprints, referee comments, revised postprints) as navigable and useful as classical peer review, or not?

My own prediction (based on nearly a quarter century of umpiring both classical peer review and open peer commentary) is that crowdsourcing will provide an excellent supplement to classical peer review but not a substitute for it. Radical implementations will simply end up re-inventing classical peer review, but on a much faster and more efficient PostGutenberg platform. We will not realize this, however, until all of the peer-reviewed literature has first been made open access. And for that it is not sufficient for Google merely to provide a platform for authors to put their unrefereed papers, because most authors don’t even put their refereed papers in their institutional repositories until it is mandated by their institutions and funders.

http://openaccess.eprints.org/index.php?/archives/1121-Crowd-Sourced-Peer-Review-Substitute-or-Supplement.html

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140825/3b246cf2/attachment.html>