[Sigmet-officers] Paper contest: appointment of reviewers

Sun Mar 13 23:27:23 EDT 2011

Dear All:

First, I will volunteer to review.

I believe Chaoqun has already set up the EasyChair system with ASIS&T for
this, so we should be able to easily implement the scoring system, once we
have agreed on one.

Cassidy

On Sat, Mar 12, 2011 at 4:52 PM, Jonathan Levitt <jonathan at levitt.net>wrote:

> *Dear all,*
>
> * *
>
> *Thanks Judit and Stasa for offering to review and for your interesting
> feedback.  *
>
> * *
>
> *I suggested six reviews per reviewer, as: (a) ISSI asked me to review six
> submissions and (b) the fewer the number of papers per reviewer the less
> liable their normalised score.  I am happy for us to opt for a four
> peepers per reviewer if we have a two stage review process; in the second
> stage the most highly rated papers from the first stage are re-reviewed.
> *
>
> * *
>
> *Judit wrote “What is the time frame for reviewing?”  According to the
> call “Authors are invited to submit manuscripts by midnight EST on Sunday,
> the 10th April 2011, to the following website ... We expect to have provided
> feedback on the submissions by the end of April 2011 and to have selected
> the winner and runner-up soon afterwards.”*
>
> * *
>
> *Stasa wrote “Is our reviewing process going to be open-ended (similar to
> Scientometrics) where one just provides a narrative, or are we going to add
> more structure (similar to JASIST) where one needs to ‘grade’ the paper on a
> number of criteria we collectively determined are the most important in
> addition to the narrative?”  To me the review process and criteria need to
> be consistent with the call.  *
>
> * *
>
> *Regarding the review process, according to the call “The contest is
> designed, not only to recognize promising student research relating to the
> SIG, but also to provide feedback from specialists in the measurement of
> information production and use. Students will receive this feedback well
> before the deadline for submissions to the ASIS&T Annual Meeting” and “There
> will be a winner, runner-up and, depending on the quantity of strong papers,
> a number of commended papers.”  These extracts indicate that the SIG will
> (a) provide feedback on student research and (b) select a winner and
> runner-up.  I suggest that in order to satisfy ‘(a)’ the reviewers will
> provide narrative feedback and in order to satisfy ‘b’ they provide an
> overall score for the paper that is then normalised.  Regarding the review
> criteria, according to the call “The reviewers will particularly reward
> well-written, original research that has potential for publication in a
> peer-reviewed journal or for presentation at a refereed conference”; this
> indicates that the review criteria should focus on (a) the quality of the
> writing and (b) the potential for publication of the research.*
>
> * *
>
> *Stasa wrote “I agree with Dietmar's suggestion that we have two reviewers
> per paper and add the third only if there are notable differences between
> the two reviewers.”  I don’t understand how a third reviewer would help us
> satisfy the stated criteria of the call and it is likely to lengthen the
> process.  Perhaps someone will explain. *
>
> * *
>
> *Stasa wrote “Who/when is going to create a template for reviewing in case
> we want to go this route?”  I think we need to agree on the criteria
> before we can create a template for reviewing.  I found my reviewer
> template for ISSI on the Easuchair system; I presume SIG/MET can arrange
> something similar.  Judit, do you know how the reviewer template was
> arranged for ISSI?*
>
> * *
>
> *Bes regards,*
>
> *Jonathan.*
>
> --- On *Sat, 12/3/11, Stasa Milojevic <smilojev at indiana.edu>* wrote:
>
>
> From: Stasa Milojevic <smilojev at indiana.edu>
>
> Subject: Re: [Sigmet-officers] Paper contest: appointment of reviewers
> To: sigmet-officers at mail.asis.org
> Date: Saturday, 12 March, 2011, 13:37
>
>
> Dear all,
>
> I am willing to serve as a fixed reviewer. I agree with Judit that six
> submissions per reviewer is too much.I agree with Dietmar's suggestion that
> we have two reviewers per paper and add the third only if there are notable
> differences between the two reviewers.
>
> Also, is our reviewing process going to be open-ended (similar to
> Scientometrics) where one just provides a narrative, or are we going to add
> more structure (similar to JASIST) where one needs to "grade" the paper on a
> number of criteria we collectively determined are the most important in
> addition to the narrative? Who/when is going to create a template for
> reviewing in case we want to go this route?
>
> Best,
> Stasa
>
> On 3/12/2011 7:17 AM, Judit Bar-Ilan wrote:
>
> Dear All,
>
> I am willing to be a fixed reviewer, but six submissions per reviewer is a
> bit too much. Four papers per reviewer is much more reasonable. So please
> rethink the process.
> Also, what is the time frame for reviewing?
>
> Regards
> Judit
>
> On Sat, Mar 12, 2011 at 12:00 PM, <sigmet-officers-request at asis.org<http://uk.mc12.mail.yahoo.com/mc/compose?to=sigmet-officers-request@asis.org>
> > wrote:
>
> Send Sigmet-officers mailing list submissions to
>        sigmet-officers at mail.asis.org<http://uk.mc12.mail.yahoo.com/mc/compose?to=sigmet-officers@mail.asis.org>
>
> To subscribe or unsubscribe via the World Wide Web, visit
>        http://mail.asis.org/mailman/listinfo/sigmet-officers
> or, via email, send a message with subject or body 'help' to
>        sigmet-officers-request at mail.asis.org<http://uk.mc12.mail.yahoo.com/mc/compose?to=sigmet-officers-request@mail.asis.org>
>
> You can reach the person managing the list at
>        sigmet-officers-owner at mail.asis.org<http://uk.mc12.mail.yahoo.com/mc/compose?to=sigmet-officers-owner@mail.asis.org>
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Sigmet-officers digest..."
>
>
> Today's Topics:
>
>   1. Re: Paper contest ? appointment of reviewers (Jonathan Levitt)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 11 Mar 2011 15:55:50 -0800 (PST)
> From: Jonathan Levitt <jonathan at levitt.net<http://uk.mc12.mail.yahoo.com/mc/compose?to=jonathan@levitt.net>
> >
> Subject: Re: [Sigmet-officers] Paper contest ? appointment of
>        reviewers
> To: sigmet-officers at asis.org<http://uk.mc12.mail.yahoo.com/mc/compose?to=sigmet-officers@asis.org>
> Message-ID: <86400.85547.qm at web1205.biz.mail.gq1.yahoo.com<http://uk.mc12.mail.yahoo.com/mc/compose?to=86400.85547.qm@web1205.biz.mail.gq1.yahoo.com>
> >
> Content-Type: text/plain; charset="utf-8"
>
> Hi all,
> ?
> Following on from Diermar?s posting, here are my initial suggestions
> regarding reviewers:
> (a)?????? Any SIG/MET officer with a Ph.D. qualifies to be a reviewer,
> irrespective of whether they have had previous experience of reviewing.
> (b)????? Establish how many qualifying SIG/MET officers are willing to
> review, before asking for external reviewers.
> (c)?????? Work out how many other people to ask to review. ?This is not
> easy as we don?t know in advance how many papers will be submitted.? However
> we can adjust the reviewing process to the number of papers submitted.?
> (d)????? One way of adjusting the reviewing process to the number of
> submissions is to have three reviewers per paper if we receive a small
> number of submissions and two reviewers per paper if we receive a small
> number of submissions. Another way of adjusting the reviewing process is for
> some reviewers to only be deployed if needed.
> (e)?????? I would be happy to be an ?optional? reviewer, i.e., to review if
> we receive a large number of papers, but not if we receive a small number of
> papers.? I suggest we quantify how many SIG/MET officers volunteering to
> review are willing to be ?optional? reviewers, before asking for external
> reviewers.
> (f)??????? Here is an example of how the reviewing process could be
> adjusted.? Suppose we have 8 reviewers, 5 ?fixed? reviewers and 3 ?optional?
> reviewers (willing to review if needed).? If we receive 10 papers, then we
> could send each of the 5 fixed reviewers 6 papers to review (30 reviews, so
> 3 reviewers per paper).? If we receive 24 papers, we could send each of the
> 8 reviewers 6 papers to review (48 reviews, so 2 reviewers per paper).?
> (g)????? The simplest way of selecting the winner and runner-up is to add
> the marks of the reviewers. ?However this is very flawed, as reviewers could
> vary enormously in the way the generosity of their marking.? ?One way of
> reducing the vagaries of markers, is to add the normalized marks (e.g., the
> number of marks divided by the average mark of the reviewer).? A more
> complicated, but I think fairer, way is to identify the strongest five or
> six articles and subject these to a second stage review.
> ?
> Please let me know what you think.
> Jonathan.
>
> --- On Wed, 9/3/11, Dietmar Wolfram <dwolfram at uwm.edu<http://uk.mc12.mail.yahoo.com/mc/compose?to=dwolfram@uwm.edu>>
> wrote:
>
>
> From: Dietmar Wolfram <dwolfram at uwm.edu<http://uk.mc12.mail.yahoo.com/mc/compose?to=dwolfram@uwm.edu>
> >
> Subject: Re: [Sigmet-officers] Paper contest ? appointment of reviewers
> To: "Jonathan Levitt" <jonathan at levitt.net<http://uk.mc12.mail.yahoo.com/mc/compose?to=jonathan@levitt.net>
> >
> Cc: sigmet-officers at asis.org<http://uk.mc12.mail.yahoo.com/mc/compose?to=sigmet-officers@asis.org>
> Date: Wednesday, 9 March, 2011, 22:46
>
>
>
> #yiv1803289635 p {margin:0;}
>
>
> Depending on the number of papers and the number of reviews per paper, we
> may only need a handful of reviewers beyond the officers, which could be
> solicited from the ISSI community, as Jonathan suggests. Would two reviews
> per submission suffice? Three reviewers are good to avoid split decisions,
> but that could add more work. Perhaps a third reviewer could be added only
> in those cases?where there are?notable differences between reviewers.
>
> Dietmar
>
>
>
>
>
>
> From: "Jonathan Levitt" <jonathan at levitt.net<http://uk.mc12.mail.yahoo.com/mc/compose?to=jonathan@levitt.net>
> >
> To: sigmet-officers at asis.org<http://uk.mc12.mail.yahoo.com/mc/compose?to=sigmet-officers@asis.org>
> Sent: Wednesday, March 9, 2011 12:47:57 PM
> Subject: [Sigmet-officers] Paper contest ? appointment of reviewers
>
>
>
>
>
>
> Hi,
>
> ?
>
> I think we should start discussing the appointment of reviewers, as I think
> we should ask the proposed reviewers well before April 10. Reviewing could
> be done by suitably qualified SIG/MET
>
> Officers and/or by reviewers for Infometric conferences (such as ISSI).?
> Any thoughts?
>
> ?
>
> Jonathan.
>
> ?
> _______________________________________________
> Sigmet-officers mailing list
> Sigmet-officers at mail.asis.org<http://uk.mc12.mail.yahoo.com/mc/compose?to=Sigmet-officers@mail.asis.org>
> http://mail.asis.org/mailman/listinfo/sigmet-officers
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mail.asis.org/pipermail/sigmet-officers/attachments/20110311/e3df66da/attachment-0001.html
>
> ------------------------------
>
> _______________________________________________
> Sigmet-officers mailing list
> Sigmet-officers at mail.asis.org<http://uk.mc12.mail.yahoo.com/mc/compose?to=Sigmet-officers@mail.asis.org>
> http://mail.asis.org/mailman/listinfo/sigmet-officers
>
>
> End of Sigmet-officers Digest, Vol 5, Issue 4
> *********************************************
>
>
>
>
> --
> Judit Bar-Ilan
> Head of Department
> Department of Information Science
> Bar-Ilan University, Ramat Gan, 52900, Israel
> Tel: 972-3-5318351 Fax: 972-3-7384027
> email: barilaj at mail.biu.ac.il<http://uk.mc12.mail.yahoo.com/mc/compose?to=barilaj@mail.biu.ac.il>
>
>
> _______________________________________________
> Sigmet-officers mailing listSigmet-officers at mail.asis.org <http://uk.mc12.mail.yahoo.com/mc/compose?to=Sigmet-officers@mail.asis.org>http://mail.asis.org/mailman/listinfo/sigmet-officers
>
>
>
> -----Inline Attachment Follows-----
>
>
> _______________________________________________
> Sigmet-officers mailing list
> Sigmet-officers at mail.asis.org<http://uk.mc12.mail.yahoo.com/mc/compose?to=Sigmet-officers@mail.asis.org>
> http://mail.asis.org/mailman/listinfo/sigmet-officers
>
>
> _______________________________________________
> Sigmet-officers mailing list
> Sigmet-officers at mail.asis.org
> http://mail.asis.org/mailman/listinfo/sigmet-officers
>
>

-- 
Cassidy R. Sugimoto, PhD
Assistant Professor
School of Library and Information Science
Indiana University Bloomington
http://ella.slis.indiana.edu/~sugimoto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/sigmet-officers/attachments/20110313/a1a34557/attachment.html 