[Sigmet-officers] scores vs ranks

Tue Mar 22 14:15:35 EDT 2011

Dear all,

Following on from your feedback, as a reviewer, I too would prefer scores over ranking.  But the person processing the scores may feel very differently, as it will require more work to process, unless we do not normalise the scares.  Though less work, scares that are not normalise are likely to have much reduced validity.

SIG III used to add the reviewers’ scores and publish the winning article in a journal.  Two very generous markers were paired together.  The winning article was regarded as so poor by the editor of the journal that she refused to publish it.    The following year we introduced normalisation.

On the basis of this experience of SIG III, if we go for scores rather than ranking, I suggest that we normalise.  I think that an easy normalisation criterion is dividing each reviewer’s score by his/her mean score, but I am not sure how valid this would be with only three papers per reviewer.   I cannot offer to spend the time on normalising.  Who is able and willing to take on this extra work of normalising?  It will need to be done quickly if there is not an undue delay in picking a winner and runner-up. 

Best regards,
Jonathan.

--- On Tue, 22/3/11, Cassidy Sugimoto <cassidysugimoto at gmail.com> wrote:

From: Cassidy Sugimoto <cassidysugimoto at gmail.com>
Subject: Re: [Sigmet-officers] Review criteria
To: "Stasa Milojevic" <smilojev at indiana.edu>
Cc: sigmet-officers at mail.asis.org
Date: Tuesday, 22 March, 2011, 17:09

I agree with this sentiment (absolute over ranking).

On Tue, Mar 22, 2011 at 12:34 PM, Stasa Milojevic <smilojev at indiana.edu> wrote:

Dear Jonathan,

I like the criteria you've suggested. I would also prefer the reviewers to give absolute scores rather than rank the papers they review. The ranking can introduce bias. Namely, one reviewer may happen to get the best 3 papers. However, since he/she will rank them, we might consider only the first one for the award, when indeed all three would qualify. On the other hand, another reviewer might receive the 3 papers that would be at the bottom of the list. However, one of them will be ranked the best, and might be considered for the award while not deserving.

Best regards,
Stasa

On 3/22/11 12:24 PM, Judit Bar-Ilan wrote: 

Dear Jonathan,

The first two evaluation criteria (overall evaluation (-3 to 3) and reviewer confidence (0 to 4: null, low, medium, high and expert) are built in and cannot be changed, and I don't think that you can submit a review to the system without marking these - possible they have a default value, but I don't remember this). Of course we can ignore these, and add our criteria on a different scale (like ISSI2011 did, all the other criteria were on a scale from 1 to 5). I am not sure, but I think that there is a way to send the scores to the authors - Chaoqun can probably test this. 
The criteria you suggested seem fine to me. I'd rather give absolute scores than rank the papers I review. 

Regards,
Judit

On Tue, Mar 22, 2011 at 5:15 PM, <sigmet-officers-request at asis.org> wrote:

Send Sigmet-officers mailing list submissions to
       sigmet-officers at mail.asis.org

To subscribe or unsubscribe via the World Wide Web, visit
       http://mail.asis.org/mailman/listinfo/sigmet-officers
or, via email, send a message with subject or body 'help' to
       sigmet-officers-request at mail.asis.org

You can reach the person managing the list at
       sigmet-officers-owner at mail.asis.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Sigmet-officers digest..."

Today's Topics:

  1. Paper contest: review criteria (Jonathan Levitt)
  2. Setting myself to no mail (Richard Hill)

----------------------------------------------------------------------

Message: 1
Date: Tue, 22 Mar 2011 08:01:45 -0700 (PDT)
From: Jonathan Levitt <jonathan at levitt.net>
Subject: [Sigmet-officers] Paper contest: review criteria
To: sigmet-officers at asis.org
Message-ID: <484740.13463.qm at web1206.biz.mail.gq1.yahoo.com>
Content-Type: text/plain; charset="utf-8"

Dear all,
?
Thanks very much Judit for letting us know that is very easy to set up evaluation criteria in Easy Chair and for your summary of the ISSI criteria. ??I have two comments regarding the ISSI criteria that seem relevant to our criteria:
(a)?????? The reviewers were asked to do work that was not sent to those who submitted or used directly in the evaluation. Specifically the reviewers needed to grade Originality, Quality of methodology/treatment, Validity of claims and interpretation, Integration into prior art and Quality of writing.? These grades were not sent to me on my submissions and seem only indirectly relevant to decision as to whether to accept the paper.
(b)????? Having three different grading scales (-3 to 3), (0 to 4) and (1 to 5) seems to add to the workload of the reviewer and seems more prone to error than using one grading scale.
?
To recap, according to the call ?The reviewers will particularly reward well-written, original research that has potential for publication in a peer-reviewed journal or for presentation at a refereed conference.?
?
I suggest that reviewers provide:
(1)????? A narrative on the quality of the writing and the potential for publication of the research and/or presentation at ASIST
(2)????? A score on the quality of the writing.
(3)????? A score on the potential for publication of the research and/or presentation at ASIST.
(4)????? Possibly a score on the reviewer?s confidence.
?
One possibility for (2) and (3) is for the reviewers to rank the papers they receive.? For example when assessing quality of the writing reviewers allocate 5 for the highest ranked paper, 4 for the second highest ranked paper, down to 2 for the fourth highest ranked paper.? This way the scores would already be normalised and we could evaluate who goes to the next round, by for each paper: (a) for each of criteria (2) and (3) adding the reviewer rankings, (b) forming an overall evaluation from the totals on criteria (2) and (3) (e.g., the weighted sum) and (c) selecting the four papers with the heist overall ranking for the second stage review.
?
What do you think?
?
Best regards,
Jonathan.

--- On Mon, 14/3/11, Judit Bar-Ilan <barilaj at mail.biu.ac.il> wrote:

From: Judit Bar-Ilan <barilaj at mail.biu.ac.il>
Subject: Re: [Sigmet-officers] Paper contest: appointment of reviewers
To: sigmet-officers at asis.org
Date: Monday, 14 March, 2011, 6:30

Dear All,

It is very easy to set up evaluation criteria in Easy Chair. Overall evaluation (on a scale of -3 to 3) and reviewer's confidence (on a scale of 0 to 4) are built in, and so are two textboxes, one for comments to the authors and one for comments to the other program committee members. Instead of a filling in the textbox it is possible to upload a file with the comments . Additional rating criteria can be added easily. For the ISSI conference these were:
Significance of problem, Originality, Quality of methodology/treatment, Validity of claims and interpretation, Integration into prior art, Quality of writing and Overall assessment - all of these on a scale of 1 to 5.

Obviously for the authors the free text narrative is much more important, but for deciding on the winner(s), scoring might be helpful, although I often find it difficult to assign scores to the evaluation criteria.

For the paper contest we are supposed to give more detailed comments than for the papers submitted to ISSI conference (some of my co-reviewers for ISSI have not commented at all, or wrote 1-2 sentences), so I still think that seriously reviewing 6 papers per reviewer is too much.

Regards,
Judit

On Sat, Mar 12, 2011 at 11:52 PM, <sigmet-officers-request at asis.org> wrote:

Send Sigmet-officers mailing list submissions to
? ? ? ?sigmet-officers at mail.asis.org

To subscribe or unsubscribe via the World Wide Web, visit
? ? ? ?http://mail.asis.org/mailman/listinfo/sigmet-officers
or, via email, send a message with subject or body 'help' to
? ? ? ?sigmet-officers-request at mail.asis.org

You can reach the person managing the list at
? ? ? ?sigmet-officers-owner at mail.asis.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Sigmet-officers digest..."

Today's Topics:

? 1. Re: Paper contest: appointment of reviewers (Jonathan Levitt)

-
-- 
Judit Bar-Ilan
Head of Department
Department of Information Science
Bar-Ilan University, Ramat Gan, 52900, Israel
Tel: 972-3-5318351 Fax: 972-3-7384027
email: barilaj at mail.biu.ac.il

_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers

_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers

-- 
Cassidy R. Sugimoto, PhD 
Assistant Professor 
School of Library and Information Science 
Indiana University Bloomington 
http://ella.slis.indiana.edu/~sugimoto 

-----Inline Attachment Follows-----

_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/sigmet-officers/attachments/20110322/393123a4/attachment.html 