[Sigmet-officers] Assignment of papers - suggested guidelines

Sun Apr 3 08:24:22 EDT 2011

Dear all,

Dietmar thanks very much for offering to jointly assign with me the submissions.  I think this is an excellent suggestion.  Assignment by the bidding system or the full list of officers is likely to result in delays.

In order to avoid delays, could we reach a consensus on the guidelines to follow before April 10?  I suggest: (a) We deploy all the externals reviewers and as many as possible of the internal reviewers – any external not deployed might be less willing to review next year, (b) Reviewing is in two stages, with the second stage being conducted by our most experienced internal reviewers (preferably Dietmar and Judit) – selecting some externasl for the second stage might offend other externals, and (c) We try to match the papers to the reviewers’ expertise and avoid asking people to review papers from their own institution.

Jonathan.

--- On Thu, 31/3/11, Dietmar Wolfram <dwolfram at uwm.edu> wrote:

From: Dietmar Wolfram <dwolfram at uwm.edu>
Subject: Re: [Sigmet-officers] Update on the paper contest
To: "Jonathan Levitt" <jonathan at levitt.net>
Cc: "SIG MET" <sigmet-officers at asis.org>
Date: Thursday, 31 March, 2011, 18:49

#yiv985505256 p {margin:0;}

I don't know if some criteria may be left blank. If originality assessment might be problematic, then perhaps a criterion that addresses the significance of the research problem could be used instead. You could have two well-crafted submissions--but if one is on an interesting and novel topic and the other is on a perennial topic that has been studied to death, it would be nice to acknowledge the submission that makes the larger contribution.

A bidding feature on a conference submission system allows reviewers to see submission abstracts and then indicate which of the submissions they would be interested in reviewing. Assignments are then based on the expressed interest. I'd be willing to join you in assigning the submissions to reviewers if no one else has done so already. 

Dietmar

From: "Jonathan Levitt" <jonathan at levitt.net>
To: "SIG MET" <sigmet-officers at asis.org>
Sent: Thursday, March 31, 2011 10:57:59 AM
Subject: Re: [Sigmet-officers] Update on the paper contest

Hi, 

Thanks Dietmar for your suggestions.  I have given my feedback on Cassidy’s suggestion in my last posting.  However I can amplify.  For ISSI I found it hard to assess the originality of a submission outside my core area (on on Web link analysis).  For my part I would not like to undertake additional work that is not central to our remit.  But, given that both you and Cassidy want additional criteria, I am content to include them, provided that reviewers could choose not to score for these additional criteria.  Is this possible?  

Could you pleases clarify what is a bidding feature?  It is important for things to movie quickly, especially as one reviewer has asked to be sent the papers as early as possible.  I agree that we need twp officers to be involved in this process, and volunteer to be one of these officers.  Could someone else who will give this matter high priority please volunteer to join me in this process? 

Thanks, 

Jonathan. 

--- On Thu, 31/3/11, Dietmar Wolfram <dwolfram at uwm.edu> wrote:

From: Dietmar Wolfram <dwolfram at uwm.edu>
Subject: Re: [Sigmet-officers] Update on the paper contest
To: "Cassidy Sugimoto" <cassidysugimoto at gmail.com>
Cc: "SIG MET" <sigmet-officers at asis.org>
Date: Thursday, 31 March, 2011, 14:19

#yiv985505256  p {margin:0;}

I also like the idea of including criteria for quality and originality. 

It looks like EasyChair does not have a bidding feature. Therefore, to keep the assignment process manageable, I suggest having two officers perform this task. Having more involved could slow things down. Those interested could let Jonathan know. 

Dietmar

From: "Cassidy Sugimoto" <cassidysugimoto at gmail.com>
Cc: "SIG MET" <sigmet-officers at asis.org>
Sent: Wednesday, March 30, 2011 11:20:56 PM
Subject: Re: [Sigmet-officers] Update on the paper contest

I know we don't want too many evaluation criteria, but I would also like to see components for the quality of the methods and the originality of the research...(and maybe "importance" or research or some other word for assessing contribution or potential to advance knowledge...)  Just my two cents--feel free to disregard.

On Wed, Mar 30, 2011 at 3:56 PM, Jonathan Levitt <jonathan at levitt.net> wrote:

Dear all, 

Of the ten potential reviewers, four have accepted (Kevin Boyack, Katherine McCain, Ronald Roussueau and Mike Thelwall), three declined and three have not yet replied. 

I suggest that we finalise the reviewing criteria well before the deadline.  On the basis of previous discussions, I suggest in addition to the default criteria (Overall evaluation, -3 to 3; 

Reviewer confidence, 0 to 4) we have the following criteria: 

(1)      Potential for publication (1 to 5). 

(2)      Quality of the writing (1 to 5). 

(3)      Comments on potential for publication and quality of writing. 

As the reviewers prefer an absolute score, I suggest we go for it.  We don’t have anyone to normalise, but hopefully the results will not be too skewed by not normalising. 

I have two questions: 

(a)       How are the papers assigned to reviewers? 

(b)      Is there any way of making sure that   reviewers receive their papers ASAP. 

Judit wrote “I am not sure, but I think that there is a way to send the scores to the authors - Chaoqun can probably test this.”   Chaoqun could you please find out and also how many papers have already been submitted. 

Best regards, 

Jonathan. 

_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers

-- 
Cassidy R. Sugimoto, PhD 
Assistant Professor 
School of Library and Information Science 
Indiana University Bloomington 
http://ella.slis.indiana.edu/~sugimoto 

_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers

-----Inline Attachment Follows-----

_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers

_______________________________________________
Sigmet-officers mailing list
Sigmet-officers at mail.asis.org
http://mail.asis.org/mailman/listinfo/sigmet-officers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/sigmet-officers/attachments/20110403/a9d67aff/attachment.html 