[Sigmet-officers] Paper contest: appointment of reviewers
Stasa Milojevic
smilojev at indiana.edu
Sat Mar 12 08:37:37 EST 2011
Dear all,
I am willing to serve as a fixed reviewer. I agree with Judit that six
submissions per reviewer is too much.I agree with Dietmar's suggestion
that we have two reviewers per paper and add the third only if there are
notable differences between the two reviewers.
Also, is our reviewing process going to be open-ended (similar to
Scientometrics) where one just provides a narrative, or are we going to
add more structure (similar to JASIST) where one needs to "grade" the
paper on a number of criteria we collectively determined are the most
important in addition to the narrative? Who/when is going to create a
template for reviewing in case we want to go this route?
Best,
Stasa
On 3/12/2011 7:17 AM, Judit Bar-Ilan wrote:
> Dear All,
>
> I am willing to be a fixed reviewer, but six submissions per reviewer
> is a bit too much. Four papers per reviewer is much more reasonable.
> So please rethink the process.
> Also, what is the time frame for reviewing?
>
> Regards
> Judit
>
> On Sat, Mar 12, 2011 at 12:00 PM, <sigmet-officers-request at asis.org
> <mailto:sigmet-officers-request at asis.org>> wrote:
>
> Send Sigmet-officers mailing list submissions to
> sigmet-officers at mail.asis.org <mailto:sigmet-officers at mail.asis.org>
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mail.asis.org/mailman/listinfo/sigmet-officers
> or, via email, send a message with subject or body 'help' to
> sigmet-officers-request at mail.asis.org
> <mailto:sigmet-officers-request at mail.asis.org>
>
> You can reach the person managing the list at
> sigmet-officers-owner at mail.asis.org
> <mailto:sigmet-officers-owner at mail.asis.org>
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Sigmet-officers digest..."
>
>
> Today's Topics:
>
> 1. Re: Paper contest ? appointment of reviewers (Jonathan Levitt)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 11 Mar 2011 15:55:50 -0800 (PST)
> From: Jonathan Levitt <jonathan at levitt.net
> <mailto:jonathan at levitt.net>>
> Subject: Re: [Sigmet-officers] Paper contest ? appointment of
> reviewers
> To: sigmet-officers at asis.org <mailto:sigmet-officers at asis.org>
> Message-ID: <86400.85547.qm at web1205.biz.mail.gq1.yahoo.com
> <mailto:86400.85547.qm at web1205.biz.mail.gq1.yahoo.com>>
> Content-Type: text/plain; charset="utf-8"
>
> Hi all,
> ?
> Following on from Diermar?s posting, here are my initial
> suggestions regarding reviewers:
> (a)?????? Any SIG/MET officer with a Ph.D. qualifies to be a
> reviewer, irrespective of whether they have had previous
> experience of reviewing.
> (b)????? Establish how many qualifying SIG/MET officers are
> willing to review, before asking for external reviewers.
> (c)?????? Work out how many other people to ask to review. ?This
> is not easy as we don?t know in advance how many papers will be
> submitted.? However we can adjust the reviewing process to the
> number of papers submitted.?
> (d)????? One way of adjusting the reviewing process to the number
> of submissions is to have three reviewers per paper if we receive
> a small number of submissions and two reviewers per paper if we
> receive a small number of submissions. Another way of adjusting
> the reviewing process is for some reviewers to only be deployed if
> needed.
> (e)?????? I would be happy to be an ?optional? reviewer, i.e., to
> review if we receive a large number of papers, but not if we
> receive a small number of papers.? I suggest we quantify how many
> SIG/MET officers volunteering to review are willing to be
> ?optional? reviewers, before asking for external reviewers.
> (f)??????? Here is an example of how the reviewing process could
> be adjusted.? Suppose we have 8 reviewers, 5 ?fixed? reviewers and
> 3 ?optional? reviewers (willing to review if needed).? If we
> receive 10 papers, then we could send each of the 5 fixed
> reviewers 6 papers to review (30 reviews, so 3 reviewers per
> paper).? If we receive 24 papers, we could send each of the 8
> reviewers 6 papers to review (48 reviews, so 2 reviewers per paper).?
> (g)????? The simplest way of selecting the winner and runner-up is
> to add the marks of the reviewers. ?However this is very flawed,
> as reviewers could vary enormously in the way the generosity of
> their marking.? ?One way of reducing the vagaries of markers, is
> to add the normalized marks (e.g., the number of marks divided by
> the average mark of the reviewer).? A more complicated, but I
> think fairer, way is to identify the strongest five or six
> articles and subject these to a second stage review.
> ?
> Please let me know what you think.
> Jonathan.
>
> --- On Wed, 9/3/11, Dietmar Wolfram <dwolfram at uwm.edu
> <mailto:dwolfram at uwm.edu>> wrote:
>
>
> From: Dietmar Wolfram <dwolfram at uwm.edu <mailto:dwolfram at uwm.edu>>
> Subject: Re: [Sigmet-officers] Paper contest ? appointment of
> reviewers
> To: "Jonathan Levitt" <jonathan at levitt.net
> <mailto:jonathan at levitt.net>>
> Cc: sigmet-officers at asis.org <mailto:sigmet-officers at asis.org>
> Date: Wednesday, 9 March, 2011, 22:46
>
>
>
> #yiv1803289635 p {margin:0;}
>
>
> Depending on the number of papers and the number of reviews per
> paper, we may only need a handful of reviewers beyond the
> officers, which could be solicited from the ISSI community, as
> Jonathan suggests. Would two reviews per submission suffice? Three
> reviewers are good to avoid split decisions, but that could add
> more work. Perhaps a third reviewer could be added only in those
> cases?where there are?notable differences between reviewers.
>
> Dietmar
>
>
>
>
>
>
> From: "Jonathan Levitt" <jonathan at levitt.net
> <mailto:jonathan at levitt.net>>
> To: sigmet-officers at asis.org <mailto:sigmet-officers at asis.org>
> Sent: Wednesday, March 9, 2011 12:47:57 PM
> Subject: [Sigmet-officers] Paper contest ? appointment of reviewers
>
>
>
>
>
>
> Hi,
>
> ?
>
> I think we should start discussing the appointment of reviewers,
> as I think we should ask the proposed reviewers well before April
> 10. Reviewing could be done by suitably qualified SIG/MET
>
> Officers and/or by reviewers for Infometric conferences (such as
> ISSI).? Any thoughts?
>
> ?
>
> Jonathan.
>
> ?
> _______________________________________________
> Sigmet-officers mailing list
> Sigmet-officers at mail.asis.org <mailto:Sigmet-officers at mail.asis.org>
> http://mail.asis.org/mailman/listinfo/sigmet-officers
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mail.asis.org/pipermail/sigmet-officers/attachments/20110311/e3df66da/attachment-0001.html
>
> ------------------------------
>
> _______________________________________________
> Sigmet-officers mailing list
> Sigmet-officers at mail.asis.org <mailto:Sigmet-officers at mail.asis.org>
> http://mail.asis.org/mailman/listinfo/sigmet-officers
>
>
> End of Sigmet-officers Digest, Vol 5, Issue 4
> *********************************************
>
>
>
>
> --
> Judit Bar-Ilan
> Head of Department
> Department of Information Science
> Bar-Ilan University, Ramat Gan, 52900, Israel
> Tel: 972-3-5318351 Fax: 972-3-7384027
> email: barilaj at mail.biu.ac.il <mailto:barilaj at mail.biu.ac.il>
>
>
> _______________________________________________
> Sigmet-officers mailing list
> Sigmet-officers at mail.asis.org
> http://mail.asis.org/mailman/listinfo/sigmet-officers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/sigmet-officers/attachments/20110312/80e98e42/attachment.html
More information about the Sigmet-officers
mailing list