[Sigmet-officers] Paper contest: appointment of reviewers
Jonathan Levitt
jonathan at levitt.net
Sat Mar 12 16:52:48 EST 2011
Dear all,
Thanks Judit and Stasa for offering to review and for your interesting feedback.
I suggested six reviews per reviewer, as: (a) ISSI asked me to review six submissions and (b) the fewer the number of papers per reviewer the less liable their normalised score. I am happy for us to opt for a four peepers per reviewer if we have a two stage review process; in the second stage the most highly rated papers from the first stage are re-reviewed.
Judit wrote “What is the time frame for reviewing?” According to the call “Authors are invited to submit manuscripts by midnight EST on Sunday, the 10th April 2011, to the following website ... We expect to have provided feedback on the submissions by the end of April 2011 and to have selected the winner and runner-up soon afterwards.”
Stasa wrote “Is our reviewing process going to be open-ended (similar to Scientometrics) where one just provides a narrative, or are we going to add more structure (similar to JASIST) where one needs to ‘grade’ the paper on a number of criteria we collectively determined are the most important in addition to the narrative?” To me the review process and criteria need to be consistent with the call.
Regarding the review process, according to the call “The contest is designed, not only to recognize promising student research relating to the SIG, but also to provide feedback from specialists in the measurement of information production and use. Students will receive this feedback well before the deadline for submissions to the ASIS&T Annual Meeting” and “There will be a winner, runner-up and, depending on the quantity of strong papers, a number of commended papers.” These extracts indicate that the SIG will (a) provide feedback on student research and (b) select a winner and runner-up. I suggest that in order to satisfy ‘(a)’ the reviewers will provide narrative feedback and in order to satisfy ‘b’ they provide an overall score for the paper that is then normalised. Regarding the review criteria, according to the call “The reviewers will particularly reward well-written, original research that has potential for publication in a
peer-reviewed journal or for presentation at a refereed conference”; this indicates that the review criteria should focus on (a) the quality of the writing and (b) the potential for publication of the research.
Stasa wrote “I agree with Dietmar's suggestion that we have two reviewers per paper and add the third only if there are notable differences between the two reviewers.” I don’t understand how a third reviewer would help us satisfy the stated criteria of the call and it is likely to lengthen the process. Perhaps someone will explain.
Stasa wrote “Who/when is going to create a template for reviewing in case we want to go this route?” I think we need to agree on the criteria before we can create a template for reviewing. I found my reviewer template for ISSI on the Easuchair system; I presume SIG/MET can arrange something similar. Judit, do you know how the reviewer template was arranged for ISSI?
Bes regards,
Jonathan.
--- On Sat, 12/3/11, Stasa Milojevic <smilojev at indiana.edu> wrote:
From: Stasa Milojevic <smilojev at indiana.edu>
Subject: Re: [Sigmet-officers] Paper contest: appointment of reviewers
To: sigmet-officers at mail.asis.org
Date: Saturday, 12 March, 2011, 13:37
Dear all,
I am willing to serve as a fixed reviewer. I agree with Judit that six submissions per reviewer is too much.I agree with Dietmar's suggestion that we have two reviewers per paper and add the third only if there are notable differences between the two reviewers.
Also, is our reviewing process going to be open-ended (similar to Scientometrics) where one just provides a narrative, or are we going to add more structure (similar to JASIST) where one needs to "grade" the paper on a number of criteria we collectively determined are the most important in addition to the narrative? Who/when is going to create a template for reviewing in case we want to go this route?
Best,
Stasa
On 3/12/2011 7:17 AM, Judit Bar-Ilan wrote:
Dear All,
I am willing to be a fixed reviewer, but six submissions per reviewer is a bit too much. Four papers per reviewer is much more reasonable. So please rethink the process.
Also, what is the time frame for reviewing?
Regards
Judit
On Sat, Mar 12, 2011 at 12:00 PM, <sigmet-officers-request at asis.org> wrote:
Send Sigmet-officers mailing list submissions to
sigmet-officers at mail.asis.org
To subscribe or unsubscribe via the World Wide Web, visit
http://mail.asis.org/mailman/listinfo/sigmet-officers
or, via email, send a message with subject or body 'help' to
sigmet-officers-request at mail.asis.org
You can reach the person managing the list at
sigmet-officers-owner at mail.asis.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Sigmet-officers digest..."
Today's Topics:
1. Re: Paper contest ? appointment of reviewers (Jonathan Levitt)
----------------------------------------------------------------------
Message: 1
Date: Fri, 11 Mar 2011 15:55:50 -0800 (PST)
From: Jonathan Levitt <jonathan at levitt.net>
Subject: Re: [Sigmet-officers] Paper contest ? appointment of
reviewers
To: sigmet-officers at asis.org
Message-ID: <86400.85547.qm at web1205.biz.mail.gq1.yahoo.com>
Content-Type: text/plain; charset="utf-8"
Hi all,
?
Following on from Diermar?s posting, here are my initial suggestions regarding reviewers:
(a)?????? Any SIG/MET officer with a Ph.D. qualifies to be a reviewer, irrespective of whether they have had previous experience of reviewing.
(b)????? Establish how many qualifying SIG/MET officers are willing to review, before asking for external reviewers.
(c)?????? Work out how many other people to ask to review. ?This is not easy as we don?t know in advance how many papers will be submitted.? However we can adjust the reviewing process to the number of papers submitted.?
(d)????? One way of adjusting the reviewing process to the number of submissions is to have three reviewers per paper if we receive a small number of submissions and two reviewers per paper if we receive a small number of submissions. Another way of adjusting the reviewing process is for some reviewers to only be deployed if needed.
(e)?????? I would be happy to be an ?optional? reviewer, i.e., to review if we receive a large number of papers, but not if we receive a small number of papers.? I suggest we quantify how many SIG/MET officers volunteering to review are willing to be ?optional? reviewers, before asking for external reviewers.
(f)??????? Here is an example of how the reviewing process could be adjusted.? Suppose we have 8 reviewers, 5 ?fixed? reviewers and 3 ?optional? reviewers (willing to review if needed).? If we receive 10 papers, then we could send each of the 5 fixed reviewers 6 papers to review (30 reviews, so 3 reviewers per paper).? If we receive 24 papers, we could send each of the 8 reviewers 6 papers to review (48 reviews, so 2 reviewers per paper).?
(g)????? The simplest way of selecting the winner and runner-up is to add the marks of the reviewers. ?However this is very flawed, as reviewers could vary enormously in the way the generosity of their marking.? ?One way of reducing the vagaries of markers, is to add the normalized marks (e.g., the number of marks divided by the average mark of the reviewer).? A more complicated, but I think fairer, way is to identify the strongest five or six articles and subject these to a second stage review.
?
Please let me know what you think.
Jonathan.
--- On Wed, 9/3/11, Dietmar Wolfram <dwolfram at uwm.edu> wrote:
From: Dietmar Wolfram <dwolfram at uwm.edu>
Subject: Re: [Sigmet-officers] Paper contest ? appointment of reviewers
To: "Jonathan Levitt" <jonathan at levitt.net>
Cc: sigmet-officers at asis.org
Date: Wednesday, 9 March, 2011, 22:46
#yiv1803289635 p {margin:0;}
Depending on the number of papers and the number of reviews per paper, we may only need a handful of reviewers beyond the officers, which could be solicited from the ISSI community, as Jonathan suggests. Would two reviews per submission suffice? Three reviewers are good to avoid split decisions, but that could add more work. Perhaps a third reviewer could be added only in those cases?where there are?notable differences between reviewers.
Dietmar
From: "Jonathan Levitt" <jonathan at levitt.net>
To: sigmet-officers at asis.org
Sent: Wednesday, March 9, 2011 12:47:57 PM
Subject: [Sigmet-officers] Paper contest ? appointment of reviewers
Hi,
?
I think we should start discussing the appointment of reviewers, as I think we should ask the proposed reviewers well before April 10. Reviewing could be done by suitably qualified SIG/MET
Officers and/or by reviewers for Infometric conferences (such as ISSI).? Any thoughts?
?
Jonathan.
?
_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/sigmet-officers/attachments/20110311/e3df66da/attachment-0001.html
------------------------------
_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers
End of Sigmet-officers Digest, Vol 5, Issue 4
*********************************************
--
Judit Bar-Ilan
Head of Department
Department of Information Science
Bar-Ilan University, Ramat Gan, 52900, Israel
Tel: 972-3-5318351 Fax: 972-3-7384027
email: barilaj at mail.biu.ac.il
_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers
-----Inline Attachment Follows-----
_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/sigmet-officers/attachments/20110312/e3f58e6c/attachment.html
More information about the Sigmet-officers
mailing list