[Sigmet-officers] Paper contest – appointment of reviewers

[Jonathan Levitt]

Hi all,
 
Following on from Diermar’s posting, here are my initial suggestions regarding reviewers:
(a)       Any SIG/MET officer with a Ph.D. qualifies to be a reviewer, irrespective of whether they have had previous experience of reviewing.
(b)      Establish how many qualifying SIG/MET officers are willing to review, before asking for external reviewers.
(c)       Work out how many other people to ask to review.  This is not easy as we don’t know in advance how many papers will be submitted.  However we can adjust the reviewing process to the number of papers submitted.  
(d)      One way of adjusting the reviewing process to the number of submissions is to have three reviewers per paper if we receive a small number of submissions and two reviewers per paper if we receive a small number of submissions. Another way of adjusting the reviewing process is for some reviewers to only be deployed if needed.
(e)       I would be happy to be an ‘optional’ reviewer, i.e., to review if we receive a large number of papers, but not if we receive a small number of papers.  I suggest we quantify how many SIG/MET officers volunteering to review are willing to be ‘optional’ reviewers, before asking for external reviewers.
(f)        Here is an example of how the reviewing process could be adjusted.  Suppose we have 8 reviewers, 5 ‘fixed’ reviewers and 3 ‘optional’ reviewers (willing to review if needed).  If we receive 10 papers, then we could send each of the 5 fixed reviewers 6 papers to review (30 reviews, so 3 reviewers per paper).  If we receive 24 papers, we could send each of the 8 reviewers 6 papers to review (48 reviews, so 2 reviewers per paper).  
(g)      The simplest way of selecting the winner and runner-up is to add the marks of the reviewers.  However this is very flawed, as reviewers could vary enormously in the way the generosity of their marking.   One way of reducing the vagaries of markers, is to add the normalized marks (e.g., the number of marks divided by the average mark of the reviewer).  A more complicated, but I think fairer, way is to identify the strongest five or six articles and subject these to a second stage review.
 
Please let me know what you think.
Jonathan.

--- On Wed, 9/3/11, Dietmar Wolfram <dwolfram at uwm.edu> wrote:


From: Dietmar Wolfram <dwolfram at uwm.edu>
Subject: Re: [Sigmet-officers] Paper contest – appointment of reviewers
To: "Jonathan Levitt" <jonathan at levitt.net>
Cc: sigmet-officers at asis.org
Date: Wednesday, 9 March, 2011, 22:46



#yiv1803289635 p {margin:0;}


Depending on the number of papers and the number of reviews per paper, we may only need a handful of reviewers beyond the officers, which could be solicited from the ISSI community, as Jonathan suggests. Would two reviews per submission suffice? Three reviewers are good to avoid split decisions, but that could add more work. Perhaps a third reviewer could be added only in those cases where there are notable differences between reviewers. 

Dietmar






From: "Jonathan Levitt" <jonathan at levitt.net>
To: sigmet-officers at asis.org
Sent: Wednesday, March 9, 2011 12:47:57 PM
Subject: [Sigmet-officers] Paper contest – appointment of reviewers






Hi, 

  

I think we should start discussing the appointment of reviewers, as I think we should ask the proposed reviewers well before April 10. Reviewing could be done by suitably qualified SIG/MET 

Officers and/or by reviewers for Infometric conferences (such as ISSI).  Any thoughts? 

  

Jonathan. 

 
_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/sigmet-officers/attachments/20110311/e3df66da/attachment.html