[Asis-l] CfP: CIKM Workshop on Living Labs for IR Evaluation - extended deadline 22 July

[Liadh Kelly]

*Living Labs for Information Retrieval Evaluation Workshop*

http://ll2013.dcu.ie


*At CIKM 2013, **November 1 2013**, **San Francisco**, **USA*

Paper and Demo Submission deadlines - Extended to July 22


*Call for Papers*

**
In the past few years, a new evaluation methodology known as *living labs *has
been proposed as a way for researchers to be able to perform in-situ
evaluation which involve and integrate users within the research process.
The basic idea of living labs for information retrieval (IR) is that rather
than individual research groups independently developing experimental
search infrastructures and gathering their own groups of test searchers for
IR evaluations, a central and shared experimental environment is developed
to facilitate the sharing of resources in a live setting (most important of
all: users). (see ll2013.dcu.ie for further details)

Living labs would offer huge benefits to the community, such as:
availability of, potentially larger, cohorts of real users and their
behaviours, e.g. querying behaviours, for experiment purposes;
cross-comparability across research centres; and greater knowledge transfer
between industry and academia, when industry partners are involved. The
need for this methodology is further amplified by the increased reliance of
IR approaches on proprietary data; living labs are a way to bridge the data
divide between academia and industry. Progress towards realising actual
living labs has nevertheless been limited. There are many challenges to be
overcome before the benefits associated with living labs for IR can be
realised, including challenges associated with living labs architecture and
design, hosting, maintenance, security, privacy, participant recruiting,
and scenarios and tasks for use development.


This workshop aims to bring together for the first time people interested
in progressing the living labs for IR evaluation methodology. An
interactive forum for researchers to share ideas and initiate
collaborations will be provided, with the explicit goal of determining
means for progressing towards living labs for IR and formulating practical
next steps for progression.


Possible use cases in this space include, but are not limited to:

· The e-commerce domain (i.e., product search and recommendation)

· The personal search space (search of personal computer files, emails, web
pages looked at, etc)

· Medical information retrieval (e.g., patients searching for medical
information on the Internet)

· Searching Wikipedia (an open-source collection with fewer privacy
concerns than, say, personal search)


Workshop topics include, but are not limited to:

· Technical challenges:

o Designing and implementing living labs architecture

o Cost of implementation

o Maintenance and adoption

o Managing living labs infrastructure

· Evaluation challenges:

o Evaluation framework and methodology

o Evaluation measures

· Privacy and security:

o Hosting data on secure server

o Gaining subjects trust

o Coping with individuals need for privacy

o Alternates when individuals will not share their data

· Legal and ethical issues:

o User consent

o Ethics approval

o Legalities regarding release of data

o Trust between parties

o Copyright issues

o Commercial sensitivity of interaction data

· Practical challenges:

o Forming living labs for IR partners within the research community

o Obtaining commercial partners

o Alternates when commercial partners cannot be obtained

o Sharing of the physical resources

o Defining tasks and scenarios for evaluation purposes


 *Paper & Demo Submissions*

Short papers (4 pages), position papers (2 pages), and posters (2 pages)
describing approaches or ideas / challenges on the topics of the workshop
are invited.


Demo papers (max. 4 pages) based on an implemented system that pursues one
or more aspects relevant to the interest areas of the workshop are also
invited.


Accepted submissions will be included in the CIKM 2013 Living Labs for
Information Retrieval Evaluation Workshop proceedings, which will be
indexed in the ACM Digital Library.


Further submission details on the workshop website: http://LL2013.dcu.ie/


*Important Dates*

*July 22, 2013**:* Deadline for paper submissions (midnight Pacific
Daylight Time)

*July 22, 2013:* Deadline for demo submissions (midnight Pacific Daylight
Time)

*August 4, 2013**:* Paper and demo notification to authors

*August 11, 2013**:* Camera-ready papers due

*November 1, 2013**:* Workshop


*Further Information*

Further information is available on the workshop website
athttp://LL2013.dcu.ie/<http://ll2013.dcu.ie/>or by emailing the
workshop organisers.


*Workshop Organisers*

Krisztian Balog - University of Stavanger, Norway (krisztian.balog (at)
uis.no)

David Elsweiler - University of Regensburg, Germany (david (at)
elsweiler.co.uk)

Evangelos Kanoulas - Google, Switzerland (ekanoulas (at) gmail.com)

Liadh Kelly - Dublin City University, Ireland (liadh.kelly (at)
computing.dcu.ie)

Mark Smucker - University of Waterloo, Canada (msmucker (at) uwaterloo.ca)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/asis-l/attachments/20130620/8877cc55/attachment-0001.html