[Sigmet-officers] Review criteria

Tue Mar 22 13:09:53 EDT 2011

I agree with this sentiment (absolute over ranking).

On Tue, Mar 22, 2011 at 12:34 PM, Stasa Milojevic <smilojev at indiana.edu>wrote:

>  Dear Jonathan,
>
> I like the criteria you've suggested. I would also prefer the reviewers to
> give absolute scores rather than rank the papers they review. The ranking
> can introduce bias. Namely, one reviewer may happen to get the best 3
> papers. However, since he/she will rank them, we might consider only the
> first one for the award, when indeed all three would qualify. On the other
> hand, another reviewer might receive the 3 papers that would be at the
> bottom of the list. However, one of them will be ranked the best, and might
> be considered for the award while not deserving.
>
> Best regards,
> Stasa
>
>
> On 3/22/11 12:24 PM, Judit Bar-Ilan wrote:
>
> Dear Jonathan,
>
> The first two evaluation criteria (overall evaluation (-3 to 3) and
> reviewer confidence (0 to 4: null, low, medium, high and expert) are built
> in and cannot be changed, and I don't think that you can submit a review to
> the system without marking these - possible they have a default value, but I
> don't remember this). Of course we can ignore these, and add our criteria on
> a different scale (like ISSI2011 did, all the other criteria were on a scale
> from 1 to 5). I am not sure, but I think that there is a way to send the
> scores to the authors - Chaoqun can probably test this.
> The criteria you suggested seem fine to me. I'd rather give absolute scores
> than rank the papers I review.
>
> Regards,
> Judit
>
> On Tue, Mar 22, 2011 at 5:15 PM, <sigmet-officers-request at asis.org> wrote:
>
>> Send Sigmet-officers mailing list submissions to
>>        sigmet-officers at mail.asis.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>        http://mail.asis.org/mailman/listinfo/sigmet-officers
>> or, via email, send a message with subject or body 'help' to
>>        sigmet-officers-request at mail.asis.org
>>
>> You can reach the person managing the list at
>>        sigmet-officers-owner at mail.asis.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Sigmet-officers digest..."
>>
>>
>> Today's Topics:
>>
>>   1. Paper contest: review criteria (Jonathan Levitt)
>>   2. Setting myself to no mail (Richard Hill)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Tue, 22 Mar 2011 08:01:45 -0700 (PDT)
>> From: Jonathan Levitt <jonathan at levitt.net>
>> Subject: [Sigmet-officers] Paper contest: review criteria
>> To: sigmet-officers at asis.org
>> Message-ID: <484740.13463.qm at web1206.biz.mail.gq1.yahoo.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Dear all,
>> ?
>> Thanks very much Judit for letting us know that is very easy to set up
>> evaluation criteria in Easy Chair and for your summary of the ISSI criteria.
>> ??I have two comments regarding the ISSI criteria that seem relevant to our
>> criteria:
>> (a)?????? The reviewers were asked to do work that was not sent to those
>> who submitted or used directly in the evaluation. Specifically the reviewers
>> needed to grade Originality, Quality of methodology/treatment, Validity of
>> claims and interpretation, Integration into prior art and Quality of
>> writing.? These grades were not sent to me on my submissions and seem only
>> indirectly relevant to decision as to whether to accept the paper.
>> (b)????? Having three different grading scales (-3 to 3), (0 to 4) and (1
>> to 5) seems to add to the workload of the reviewer and seems more prone to
>> error than using one grading scale.
>> ?
>> To recap, according to the call ?The reviewers will particularly reward
>> well-written, original research that has potential for publication in a
>> peer-reviewed journal or for presentation at a refereed conference.?
>> ?
>> I suggest that reviewers provide:
>> (1)????? A narrative on the quality of the writing and the potential for
>> publication of the research and/or presentation at ASIST
>> (2)????? A score on the quality of the writing.
>> (3)????? A score on the potential for publication of the research and/or
>> presentation at ASIST.
>> (4)????? Possibly a score on the reviewer?s confidence.
>> ?
>> One possibility for (2) and (3) is for the reviewers to rank the papers
>> they receive.? For example when assessing quality of the writing reviewers
>> allocate 5 for the highest ranked paper, 4 for the second highest ranked
>> paper, down to 2 for the fourth highest ranked paper.? This way the scores
>> would already be normalised and we could evaluate who goes to the next
>> round, by for each paper: (a) for each of criteria (2) and (3) adding the
>> reviewer rankings, (b) forming an overall evaluation from the totals on
>> criteria (2) and (3) (e.g., the weighted sum) and (c) selecting the four
>> papers with the heist overall ranking for the second stage review.
>> ?
>> What do you think?
>> ?
>> Best regards,
>> Jonathan.
>>
>> --- On Mon, 14/3/11, Judit Bar-Ilan <barilaj at mail.biu.ac.il> wrote:
>>
>>
>> From: Judit Bar-Ilan <barilaj at mail.biu.ac.il>
>> Subject: Re: [Sigmet-officers] Paper contest: appointment of reviewers
>> To: sigmet-officers at asis.org
>> Date: Monday, 14 March, 2011, 6:30
>>
>>
>>
>> Dear All,
>>
>> It is very easy to set up evaluation criteria in Easy Chair. Overall
>> evaluation (on a scale of -3 to 3) and reviewer's confidence (on a scale of
>> 0 to 4) are built in, and so are two textboxes, one for comments to the
>> authors and one for comments to the other program committee members. Instead
>> of a filling in the textbox it is possible to upload a file with the
>> comments . Additional rating criteria can be added easily. For the ISSI
>> conference these were:
>> Significance of problem, Originality, Quality of methodology/treatment,
>> Validity of claims and interpretation, Integration into prior art, Quality
>> of writing and Overall assessment - all of these on a scale of 1 to 5.
>>
>> Obviously for the authors the free text narrative is much more important,
>> but for deciding on the winner(s), scoring might be helpful, although I
>> often find it difficult to assign scores to the evaluation criteria.
>>
>> For the paper contest we are supposed to give more detailed comments than
>> for the papers submitted to ISSI conference (some of my co-reviewers for
>> ISSI have not commented at all, or wrote 1-2 sentences), so I still think
>> that seriously reviewing 6 papers per reviewer is too much.
>>
>> Regards,
>> Judit
>>
>>
>> On Sat, Mar 12, 2011 at 11:52 PM, <sigmet-officers-request at asis.org>
>> wrote:
>>
>> Send Sigmet-officers mailing list submissions to
>> ? ? ? ?sigmet-officers at mail.asis.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> ? ? ? ?http://mail.asis.org/mailman/listinfo/sigmet-officers
>> or, via email, send a message with subject or body 'help' to
>> ? ? ? ?sigmet-officers-request at mail.asis.org
>>
>> You can reach the person managing the list at
>> ? ? ? ?sigmet-officers-owner at mail.asis.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Sigmet-officers digest..."
>>
>>
>> Today's Topics:
>>
>> ? 1. Re: Paper contest: appointment of reviewers (Jonathan Levitt)
>>
>>
>> -
>>
>  --
> Judit Bar-Ilan
> Head of Department
> Department of Information Science
> Bar-Ilan University, Ramat Gan, 52900, Israel
> Tel: 972-3-5318351 Fax: 972-3-7384027
> email: barilaj at mail.biu.ac.il
>
>
> _______________________________________________
> Sigmet-officers mailing listSigmet-officers at mail.asis.org
> http://mail.asis.org/mailman/listinfo/sigmet-officers
>
>
>
> _______________________________________________
> Sigmet-officers mailing list
> Sigmet-officers at mail.asis.org
> http://mail.asis.org/mailman/listinfo/sigmet-officers
>
>

-- 
Cassidy R. Sugimoto, PhD
Assistant Professor
School of Library and Information Science
Indiana University Bloomington
http://ella.slis.indiana.edu/~sugimoto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/sigmet-officers/attachments/20110322/cacd2dc1/attachment.html 