[Asis-l] CfP: CIKM Workshop on Living Labs for Information Retrieval Evaluation

[Liadh Kelly]

*Living Labs for Information Retrieval Evaluation Workshop*

http://ll2013.dcu.ie

*At CIKM 2013, **November 1 2013**, **San Francisco**, **USA*

Poll: should we have a living labs IR challenge using a medium-sized 
online retailer?

http://bit.ly/LL13poll

*********************************************************************

*Call for Papers*

Submission deadline: June 21

In the past few years, a new evaluation methodology known as /living 
labs /has been proposed as a way for researchers to be able to perform 
in-situ evaluation. The basic idea of living labs for information 
retrieval (IR) is that rather than individual research groups 
independently developing experimental search infrastructures and 
gathering their own groups of test searchers for IR evaluations, a 
central and shared experimental environment is developed to facilitate 
the sharing of resources.

Living labs would offer huge benefits to the community, such as: 
availability of, potentially larger, cohorts of real users and their 
behaviours, e.g. querying behaviours, for experiment purposes; 
cross-comparability across research centres; and greater knowledge 
transfer between industry and academia, when industry partners are 
involved. The need for this methodology is further amplified by the 
increased reliance of IR approaches on proprietary data; living labs are 
a way to bridge the data divide between academia and industry. Progress 
towards realising actual living labs has nevertheless been limited. 
There are many challenges to be overcome before the benefits associated 
with living labs for IR can be realised, including challenges associated 
with living labs architecture and design, hosting, maintenance, 
security, privacy, participant recruiting, and scenarios and tasks for 
use development.

This workshop aims to bring together for the first time people 
interested in progressing the living labs for IR evaluation methodology. 
An interactive forum for researchers to share ideas and initiate 
collaborations will be provided, with the explicit goal of determining 
means for progressing towards living labs for IR and formulating 
practical next steps for progression.

Possible use cases in this space include, but are not limited to:

· The e-commerce domain (i.e., product search and recommendation)

· The personal search space (search of personal computer files, emails, 
web pages looked at, etc)

· Medical information retrieval (e.g., patients searching for medical 
information on the Internet)

· Searching Wikipedia (an open-source collection with fewer privacy 
concerns than, say, personal search)

Workshop topics include, but are not limited to:

· Privacy and security:

o Hosting data on secure server

o Gaining subjects trust

o Coping with individuals need for privacy

o Alternates when individuals will not share their data

· Legal and ethical issues:

o User consent

o Ethics approval

o Legalities regarding release of data

o Trust between parties

o Copyright issues

o Commercial sensitivity of interaction data

· Technical challenges:

o Designing and implementing living labs architecture

o Cost of implementation

o Maintenance and adoption

o Managing living labs infrastructure

· Practical challenges:

o Forming living labs for IR partners within the research community

o Obtaining commercial partners

o Alternates when commercial partners cannot be obtained

o Sharing of the physical resources

o Defining tasks and scenarios for evaluation purposes


    Poll - Should we have a living labs IR challenge using a
    medium-sized online retailer?

As part of the Living Labs for Information Retrieval Evaluation 
workshop, we are considering organising a challenge in the e-commerce 
domain with the involvement of a medium-sized online retailer. The goal 
of this challenge would be to (i) allow academics to work with real 
users and data (esp. those who otherwise would have no access to such 
data) and (ii) to provide a starting point for the discussions at the 
workshop.

We will set up and run this challenge if there is sufficient interest in 
the community. So please let us know what you think on the poll at 
http://bit.ly/LL13poll.

*Paper Submissions*

The workshop is now accepting paper submissions. Short papers (4 pages), 
position papers (2 pages), and posters (2 pages) describing approaches 
or ideas / challenges on the topics of the workshop are invited. 
Submissions should be in ACM SIGS format. LaTeX and Word templates are 
available 
athttp://www.acm.org/sigs/publications/proceedings-templates(for LaTeX, 
use the "Option 2" style).

Papers should be anonymised for double blind review and submitted in pdf 
format through the EasyChair 
system<http://www.easychair.org/...>https://www.easychair.org/conferences/?conf=ll2013no 
later than midnightPacific Daylight Time on June 21, 2013.Submissions 
will be reviewed by members of the workshop program committee. Accepted 
papers will be included in the CIKM 2013 Living Labs for Information 
Retrieval Evaluation Workshop proceedings.

*Important Dates*

*June 21, 2013**:*Deadline for paper submission (midnightPacific 
Daylight Time)

**

*July 22, 2013**:*Notification to authors

**

*August 11, 2013**:*Camera-ready paper due

**

*November 1, 2013**:*Workshop

*Further Information*

Further information is available on the workshop website 
athttp://LL2013.dcu.ie/ <http://ll2013.dcu.ie/>or by emailing the 
workshop organisers.

*Workshop Organisers*

Krisztian Balog - University of Stavanger, Norway(krisztian.balog (at) 
uis.no)

David Elsweiler - University of Regensburg, Germany(david (at) 
elsweiler.co.uk)

Evangelos Kanoulas - Google, Switzerland(ekanoulas (at) gmail.com)

Liadh Kelly - Dublin City University, Ireland(liadh.kelly (at) 
computing.dcu.ie)

Mark Smucker - University of Waterloo, Canada(msmucker (at) uwaterloo.ca)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/asis-l/attachments/20130503/b200fe56/attachment-0001.html