[Sigmet-officers] Review criteria

[Judit Bar-Ilan]

Dear Jonathan,

The first two evaluation criteria (overall evaluation (-3 to 3) and reviewer
confidence (0 to 4: null, low, medium, high and expert) are built in and
cannot be changed, and I don't think that you can submit a review to the
system without marking these - possible they have a default value, but I
don't remember this). Of course we can ignore these, and add our criteria on
a different scale (like ISSI2011 did, all the other criteria were on a scale
from 1 to 5). I am not sure, but I think that there is a way to send the
scores to the authors - Chaoqun can probably test this.
The criteria you suggested seem fine to me. I'd rather give absolute scores
than rank the papers I review.

Regards,
Judit

On Tue, Mar 22, 2011 at 5:15 PM, <sigmet-officers-request at asis.org> wrote:

> Send Sigmet-officers mailing list submissions to
>        sigmet-officers at mail.asis.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>        http://mail.asis.org/mailman/listinfo/sigmet-officers
> or, via email, send a message with subject or body 'help' to
>        sigmet-officers-request at mail.asis.org
>
> You can reach the person managing the list at
>        sigmet-officers-owner at mail.asis.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Sigmet-officers digest..."
>
>
> Today's Topics:
>
>   1. Paper contest: review criteria (Jonathan Levitt)
>   2. Setting myself to no mail (Richard Hill)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 22 Mar 2011 08:01:45 -0700 (PDT)
> From: Jonathan Levitt <jonathan at levitt.net>
> Subject: [Sigmet-officers] Paper contest: review criteria
> To: sigmet-officers at asis.org
> Message-ID: <484740.13463.qm at web1206.biz.mail.gq1.yahoo.com>
> Content-Type: text/plain; charset="utf-8"
>
> Dear all,
> ?
> Thanks very much Judit for letting us know that is very easy to set up
> evaluation criteria in Easy Chair and for your summary of the ISSI criteria.
> ??I have two comments regarding the ISSI criteria that seem relevant to our
> criteria:
> (a)?????? The reviewers were asked to do work that was not sent to those
> who submitted or used directly in the evaluation. Specifically the reviewers
> needed to grade Originality, Quality of methodology/treatment, Validity of
> claims and interpretation, Integration into prior art and Quality of
> writing.? These grades were not sent to me on my submissions and seem only
> indirectly relevant to decision as to whether to accept the paper.
> (b)????? Having three different grading scales (-3 to 3), (0 to 4) and (1
> to 5) seems to add to the workload of the reviewer and seems more prone to
> error than using one grading scale.
> ?
> To recap, according to the call ?The reviewers will particularly reward
> well-written, original research that has potential for publication in a
> peer-reviewed journal or for presentation at a refereed conference.?
> ?
> I suggest that reviewers provide:
> (1)????? A narrative on the quality of the writing and the potential for
> publication of the research and/or presentation at ASIST
> (2)????? A score on the quality of the writing.
> (3)????? A score on the potential for publication of the research and/or
> presentation at ASIST.
> (4)????? Possibly a score on the reviewer?s confidence.
> ?
> One possibility for (2) and (3) is for the reviewers to rank the papers
> they receive.? For example when assessing quality of the writing reviewers
> allocate 5 for the highest ranked paper, 4 for the second highest ranked
> paper, down to 2 for the fourth highest ranked paper.? This way the scores
> would already be normalised and we could evaluate who goes to the next
> round, by for each paper: (a) for each of criteria (2) and (3) adding the
> reviewer rankings, (b) forming an overall evaluation from the totals on
> criteria (2) and (3) (e.g., the weighted sum) and (c) selecting the four
> papers with the heist overall ranking for the second stage review.
> ?
> What do you think?
> ?
> Best regards,
> Jonathan.
>
> --- On Mon, 14/3/11, Judit Bar-Ilan <barilaj at mail.biu.ac.il> wrote:
>
>
> From: Judit Bar-Ilan <barilaj at mail.biu.ac.il>
> Subject: Re: [Sigmet-officers] Paper contest: appointment of reviewers
> To: sigmet-officers at asis.org
> Date: Monday, 14 March, 2011, 6:30
>
>
>
> Dear All,
>
> It is very easy to set up evaluation criteria in Easy Chair. Overall
> evaluation (on a scale of -3 to 3) and reviewer's confidence (on a scale of
> 0 to 4) are built in, and so are two textboxes, one for comments to the
> authors and one for comments to the other program committee members. Instead
> of a filling in the textbox it is possible to upload a file with the
> comments . Additional rating criteria can be added easily. For the ISSI
> conference these were:
> Significance of problem, Originality, Quality of methodology/treatment,
> Validity of claims and interpretation, Integration into prior art, Quality
> of writing and Overall assessment - all of these on a scale of 1 to 5.
>
> Obviously for the authors the free text narrative is much more important,
> but for deciding on the winner(s), scoring might be helpful, although I
> often find it difficult to assign scores to the evaluation criteria.
>
> For the paper contest we are supposed to give more detailed comments than
> for the papers submitted to ISSI conference (some of my co-reviewers for
> ISSI have not commented at all, or wrote 1-2 sentences), so I still think
> that seriously reviewing 6 papers per reviewer is too much.
>
> Regards,
> Judit
>
>
> On Sat, Mar 12, 2011 at 11:52 PM, <sigmet-officers-request at asis.org>
> wrote:
>
> Send Sigmet-officers mailing list submissions to
> ? ? ? ?sigmet-officers at mail.asis.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> ? ? ? ?http://mail.asis.org/mailman/listinfo/sigmet-officers
> or, via email, send a message with subject or body 'help' to
> ? ? ? ?sigmet-officers-request at mail.asis.org
>
> You can reach the person managing the list at
> ? ? ? ?sigmet-officers-owner at mail.asis.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Sigmet-officers digest..."
>
>
> Today's Topics:
>
> ? 1. Re: Paper contest: appointment of reviewers (Jonathan Levitt)
>
>
> -
>
-- 
Judit Bar-Ilan
Head of Department
Department of Information Science
Bar-Ilan University, Ramat Gan, 52900, Israel
Tel: 972-3-5318351 Fax: 972-3-7384027
email: barilaj at mail.biu.ac.il
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/sigmet-officers/attachments/20110322/9a7a423b/attachment.html