[Sigmet-officers] Paper contest: number of reviewers, letter to reviewers

Thu Mar 17 11:32:16 EDT 2011

Thanks Dietmar and Cassidy for your suggestions regarding reviewers.  Thanks Dietmar for letting g me know that you had offered to review.  I am sorry I missed this in the correspondence.  If we arrange four external reviewers, then we would have nine reviewers.  The four papers per reviewer limit would limit us to evaluating a maximum of 18 entries.  What happens if we receive more entries? 

I am concerned that, without a second round of reviewing for the highest scoring few papers, the selecting of the winner and runner-up could be somewhat arbitrary.  If we are going for a second round of reviewing, we will also need reviewers for this.

Jonathan.

--- On Thu, 17/3/11, Cassidy Sugimoto <cassidysugimoto at gmail.com> wrote:

From: Cassidy Sugimoto <cassidysugimoto at gmail.com>
Subject: Re: [Sigmet-officers] Paper contest: number of reviewers, letter to reviewers
To: "Dietmar Wolfram" <dwolfram at uwm.edu>
Cc: "Jonathan Levitt" <jonathan at levitt.net>, sigmet-officers at asis.org
Date: Thursday, 17 March, 2011, 13:00

I would suggest Howard White. I'm sure he would be willing to do it.

On Thu, Mar 17, 2011 at 8:20 AM, Dietmar Wolfram <dwolfram at uwm.edu> wrote:

Jonathan,

As I mentioned earlier, I am happy to review papers. I don't think it should be necessary to contact more than four external reviewers. Perhaps a couple of ASIST members from North America (maybe Drexel and Western Ontario) and a couple from Europe or Asia could be contacted. This will help to get them involved in the SIG. 

Dietmar 

From: "Jonathan Levitt" <jonathan at levitt.net>

To: sigmet-officers at asis.org
Sent: Thursday, March 17, 2011 6:59:11 AM
Subject: [Sigmet-officers] Paper contest: number of reviewers,        letter to reviewers

Dear all, 

Thanks for your latest suggestions.  We have at least four reviewers amongst the officers and we have agreed to a maximum of four papers for reviewer. To me the most urgent matters are for us to decide how many external reviewers to enlist and for Chaoqun to contact them. 

Unfortunately we need to have enlisted reviewers before the April 10 deadline, but we won’t know how many papers to review until after the April 10 deadline (I sought to address this problem in my March 11 email).  Nor do we know in advance what percentage of the people contacted will agree to review.  Has anyone any suggestions on how many external reviewers to enlist?  

Whilst we decide on how many external reviewers to enlist, I suggest that we prepare the letter to external reviewers.  Chaoqun, could you please draft the letter to external reviewers? 

Best regards, 

Jonathan. 

--- On Mon, 14/3/11, Judit Bar-Ilan <barilaj at mail.biu.ac.il> wrote:

From: Judit Bar-Ilan <barilaj at mail.biu.ac.il>
Subject: Re: [Sigmet-officers] Paper contest: appointment of reviewers
To: sigmet-officers at asis.org
Date: Monday, 14 March, 2011, 6:30

Dear All,

It is very easy to set up evaluation criteria in Easy Chair. Overall evaluation (on a scale of -3 to 3) and reviewer's confidence (on a scale of 0 to 4) are built in, and so are two textboxes, one for comments to the authors and one for comments to the other program committee members. Instead of a filling in the textbox it is possible to upload a file with the comments . Additional rating criteria can be added easily. For the ISSI conference these were:
Significance of problem, Originality, Quality of methodology/treatment, Validity of claims and interpretation, Integration into prior art, Quality of writing and Overall assessment - all of these on a scale of 1 to 5. 

Obviously for the authors the free text narrative is much more important, but for deciding on the winner(s), scoring might be helpful, although I often find it difficult to assign scores to the evaluation criteria.

For the paper contest we are supposed to give more detailed comments than for the papers submitted to ISSI conference (some of my co-reviewers for ISSI have not commented at all, or wrote 1-2 sentences), so I still think that seriously reviewing 6 papers per reviewer is too much.

Regards,
Judit

On Sat, Mar 12, 2011 at 11:52 PM, <sigmet-officers-request at asis.org> wrote:

Send Sigmet-officers mailing list submissions to
       sigmet-officers at mail.asis.org

To subscribe or unsubscribe via the World Wide Web, visit
       http://mail.asis.org/mailman/listinfo/sigmet-officers
or, via email, send a message with subject or body 'help' to
       sigmet-officers-request at mail.asis.org

You can reach the person managing the list at
       sigmet-officers-owner at mail.asis.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Sigmet-officers digest..."

Today's Topics:

  1. Re: Paper contest: appointment of reviewers (Jonathan Levitt)

----------------------------------------------------------------------

Message: 1
Date: Sat, 12 Mar 2011 13:52:48 -0800 (PST)
From: Jonathan Levitt <jonathan at levitt.net>
Subject: Re: [Sigmet-officers] Paper contest: appointment of reviewers
To: sigmet-officers at mail.asis.org
Message-ID: <403991.8284.qm at web1206.biz.mail.gq1.yahoo.com>
Content-Type: text/plain; charset="utf-8"

Dear all,
?
Thanks Judit and Stasa for offering to review and for your interesting feedback.?
?
I suggested six reviews per reviewer, as: (a) ISSI asked me to review six submissions and (b) the fewer the number of papers per reviewer the less liable their normalised score.? I am happy for us to opt for a four peepers per reviewer if we have a two stage review process; in the second stage the most highly rated papers from the first stage are re-reviewed.?
?
Judit wrote ?What is the time frame for reviewing??? According to the call ?Authors are invited to submit manuscripts by midnight EST on Sunday, the 10th April 2011, to the following website ... We expect to have provided feedback on the submissions by the end of April 2011 and to have selected the winner and runner-up soon afterwards.?
?
Stasa wrote ?Is our reviewing process going to be open-ended (similar to Scientometrics) where one just provides a narrative, or are we going to add more structure (similar to JASIST) where one needs to ?grade? the paper on a number of criteria we collectively determined are the most important in addition to the narrative??? To me the review process and criteria need to be consistent with the call.?
?
Regarding the review process, according to the call ?The contest is designed, not only to recognize promising student research relating to the SIG, but also to provide feedback from specialists in the measurement of information production and use. Students will receive this feedback well before the deadline for submissions to the ASIS&T Annual Meeting? and ?There will be a winner, runner-up and, depending on the quantity of strong papers, a number of commended papers.? ?These extracts indicate that the SIG will (a) provide feedback on student research and (b) select a winner and runner-up. ?I suggest that in order to satisfy ?(a)? the reviewers will provide narrative feedback and in order to satisfy ?b? they provide an overall score for the paper that is then normalised.? Regarding the review criteria, according to the call ?The reviewers will particularly reward well-written, original research that has potential for publication in a
 peer-reviewed journal or for presentation at a refereed conference?; this indicates that the review criteria should focus on (a) the quality of the writing and (b) the potential for publication of the research.
?
Stasa wrote ?I agree with Dietmar's suggestion that we have two reviewers per paper and add the third only if there are notable differences between the two reviewers.? ?I don?t understand how a third reviewer would help us satisfy the stated criteria of the call and it is likely to lengthen the process.? Perhaps someone will explain.
?
Stasa wrote ?Who/when is going to create a template for reviewing in case we want to go this route??? I think we need to agree on the criteria before we can create a template for reviewing.? I found my reviewer template for ISSI on the Easuchair system; I presume SIG/MET can arrange something similar.? Judit, do you know how the reviewer template was arranged for ISSI?
?
Bes regards,
Jonathan.

-- 
Judit Bar-Ilan
Head of Department
Department of Information Science
Bar-Ilan University, Ramat Gan, 52900, Israel
Tel: 972-3-5318351 Fax: 972-3-7384027
email: barilaj at mail.biu.ac.il

-----Inline Attachment Follows-----

_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers

_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers

_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers

-- 
Cassidy R. Sugimoto, PhD 
Assistant Professor 
School of Library and Information Science 
Indiana University Bloomington 
http://ella.slis.indiana.edu/~sugimoto 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/sigmet-officers/attachments/20110317/d11de7f9/attachment.html 