[Sigmet-officers] Setting myself to no mail

Tue Mar 22 11:14:43 EDT 2011

This is well under way and since I do not need to see all this, I have set
myself to "no mail."

If there are problems, reply to me directly - rhill at asis.org

Dick

  _____  

From: sigmet-officers-bounces at asis.org
[mailto:sigmet-officers-bounces at asis.org] On Behalf Of Jonathan Levitt
Sent: Tuesday, March 22, 2011 11:02 AM
To: sigmet-officers at asis.org
Subject: [Sigmet-officers] Paper contest: review criteria

Dear all,

Thanks very much Judit for letting us know that is very easy to set up
evaluation criteria in Easy Chair and for your summary of the ISSI criteria.
I have two comments regarding the ISSI criteria that seem relevant to our
criteria:

(a)       The reviewers were asked to do work that was not sent to those who
submitted or used directly in the evaluation. Specifically the reviewers
needed to grade Originality, Quality of methodology/treatment, Validity of
claims and interpretation, Integration into prior art and Quality of
writing.  These grades were not sent to me on my submissions and seem only
indirectly relevant to decision as to whether to accept the paper.

(b)      Having three different grading scales (-3 to 3), (0 to 4) and (1 to
5) seems to add to the workload of the reviewer and seems more prone to
error than using one grading scale.

To recap, according to the call "The reviewers will particularly reward
well-written, original research that has potential for publication in a
peer-reviewed journal or for presentation at a refereed conference."

I suggest that reviewers provide:

(1)      A narrative on the quality of the writing and the potential for
publication of the research and/or presentation at ASIST

(2)      A score on the quality of the writing.

(3)      A score on the potential for publication of the research and/or
presentation at ASIST.

(4)      Possibly a score on the reviewer's confidence.

One possibility for (2) and (3) is for the reviewers to rank the papers they
receive.  For example when assessing quality of the writing reviewers
allocate 5 for the highest ranked paper, 4 for the second highest ranked
paper, down to 2 for the fourth highest ranked paper.  This way the scores
would already be normalised and we could evaluate who goes to the next
round, by for each paper: (a) for each of criteria (2) and (3) adding the
reviewer rankings, (b) forming an overall evaluation from the totals on
criteria (2) and (3) (e.g., the weighted sum) and (c) selecting the four
papers with the heist overall ranking for the second stage review.

What do you think?

Best regards,

Jonathan.

--- On Mon, 14/3/11, Judit Bar-Ilan <barilaj at mail.biu.ac.il> wrote:

From: Judit Bar-Ilan <barilaj at mail.biu.ac.il>
Subject: Re: [Sigmet-officers] Paper contest: appointment of reviewers
To: sigmet-officers at asis.org
Date: Monday, 14 March, 2011, 6:30

Dear All,

It is very easy to set up evaluation criteria in Easy Chair. Overall
evaluation (on a scale of -3 to 3) and reviewer's confidence (on a scale of
0 to 4) are built in, and so are two textboxes, one for comments to the
authors and one for comments to the other program committee members. Instead
of a filling in the textbox it is possible to upload a file with the
comments . Additional rating criteria can be added easily. For the ISSI
conference these were:
Significance of problem, Originality, Quality of methodology/treatment,
Validity of claims and interpretation, Integration into prior art, Quality
of writing and Overall assessment - all of these on a scale of 1 to 5. 

Obviously for the authors the free text narrative is much more important,
but for deciding on the winner(s), scoring might be helpful, although I
often find it difficult to assign scores to the evaluation criteria.

For the paper contest we are supposed to give more detailed comments than
for the papers submitted to ISSI conference (some of my co-reviewers for
ISSI have not commented at all, or wrote 1-2 sentences), so I still think
that seriously reviewing 6 papers per reviewer is too much.

Regards,
Judit

On Sat, Mar 12, 2011 at 11:52 PM, <sigmet-officers-request at asis.org
<http://uk.mc12.mail.yahoo.com/mc/compose?to=sigmet-officers-request@asis.or
g> > wrote:

Send Sigmet-officers mailing list submissions to
       sigmet-officers at mail.asis.org
<http://uk.mc12.mail.yahoo.com/mc/compose?to=sigmet-officers@mail.asis.org> 

To subscribe or unsubscribe via the World Wide Web, visit
       http://mail.asis.org/mailman/listinfo/sigmet-officers
or, via email, send a message with subject or body 'help' to
       sigmet-officers-request at mail.asis.org
<http://uk.mc12.mail.yahoo.com/mc/compose?to=sigmet-officers-request@mail.as
is.org> 

You can reach the person managing the list at
       sigmet-officers-owner at mail.asis.org
<http://uk.mc12.mail.yahoo.com/mc/compose?to=sigmet-officers-owner@mail.asis
.org> 

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Sigmet-officers digest..."

Today's Topics:

  1. Re: Paper contest: appointment of reviewers (Jonathan Levitt)

----------------------------------------------------------------------

Message: 1
Date: Sat, 12 Mar 2011 13:52:48 -0800 (PST)
From: Jonathan Levitt <jonathan at levitt.net
<http://uk.mc12.mail.yahoo.com/mc/compose?to=jonathan@levitt.net> >
Subject: Re: [Sigmet-officers] Paper contest: appointment of reviewers
To: sigmet-officers at mail.asis.org
<http://uk.mc12.mail.yahoo.com/mc/compose?to=sigmet-officers@mail.asis.org> 
Message-ID: <403991.8284.qm at web1206.biz.mail.gq1.yahoo.com
<http://uk.mc12.mail.yahoo.com/mc/compose?to=403991.8284.qm@web1206.biz.mail
.gq1.yahoo.com> >
Content-Type: text/plain; charset="utf-8"

Dear all,
?
Thanks Judit and Stasa for offering to review and for your interesting
feedback.?
?
I suggested six reviews per reviewer, as: (a) ISSI asked me to review six
submissions and (b) the fewer the number of papers per reviewer the less
liable their normalised score.? I am happy for us to opt for a four peepers
per reviewer if we have a two stage review process; in the second stage the
most highly rated papers from the first stage are re-reviewed.?
?
Judit wrote ?What is the time frame for reviewing??? According to the call
?Authors are invited to submit manuscripts by midnight EST on Sunday, the
10th April 2011, to the following website ... We expect to have provided
feedback on the submissions by the end of April 2011 and to have selected
the winner and runner-up soon afterwards.?
?
Stasa wrote ?Is our reviewing process going to be open-ended (similar to
Scientometrics) where one just provides a narrative, or are we going to add
more structure (similar to JASIST) where one needs to ?grade? the paper on a
number of criteria we collectively determined are the most important in
addition to the narrative??? To me the review process and criteria need to
be consistent with the call.?
?
Regarding the review process, according to the call ?The contest is
designed, not only to recognize promising student research relating to the
SIG, but also to provide feedback from specialists in the measurement of
information production and use. Students will receive this feedback well
before the deadline for submissions to the ASIS&T Annual Meeting? and ?There
will be a winner, runner-up and, depending on the quantity of strong papers,
a number of commended papers.? ?These extracts indicate that the SIG will
(a) provide feedback on student research and (b) select a winner and
runner-up. ?I suggest that in order to satisfy ?(a)? the reviewers will
provide narrative feedback and in order to satisfy ?b? they provide an
overall score for the paper that is then normalised.? Regarding the review
criteria, according to the call ?The reviewers will particularly reward
well-written, original research that has potential for publication in a
 peer-reviewed journal or for presentation at a refereed conference?; this
indicates that the review criteria should focus on (a) the quality of the
writing and (b) the potential for publication of the research.
?
Stasa wrote ?I agree with Dietmar's suggestion that we have two reviewers
per paper and add the third only if there are notable differences between
the two reviewers.? ?I don?t understand how a third reviewer would help us
satisfy the stated criteria of the call and it is likely to lengthen the
process.? Perhaps someone will explain.
?
Stasa wrote ?Who/when is going to create a template for reviewing in case we
want to go this route??? I think we need to agree on the criteria before we
can create a template for reviewing.? I found my reviewer template for ISSI
on the Easuchair system; I presume SIG/MET can arrange something similar.?
Judit, do you know how the reviewer template was arranged for ISSI?
?
Bes regards,
Jonathan.

-- 
Judit Bar-Ilan
Head of Department
Department of Information Science
Bar-Ilan University, Ramat Gan, 52900, Israel
Tel: 972-3-5318351 Fax: 972-3-7384027
email: barilaj at mail.biu.ac.il
<http://uk.mc12.mail.yahoo.com/mc/compose?to=barilaj@mail.biu.ac.il> 

-----Inline Attachment Follows-----

_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
<http://uk.mc12.mail.yahoo.com/mc/compose?to=Sigmet-officers@mail.asis.org> 
http://mail.asis.org/mailman/listinfo/sigmet-officers

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/sigmet-officers/attachments/20110322/a86c26f6/attachment.html 