[Asis-l] 2nd CfP: CIKM Living Labs for Information Retrieval Evaluation Workshop

[Liadh Kelly]

*Living Labs for Information Retrieval Evaluation Workshop*

http://ll2013.dcu.ie



*At CIKM 2013, **November 1 2013**, **San Francisco**, **USA*



Poll: should we have a living labs IR challenge using a medium-sized online
retailer?

http://bit.ly/LL13poll



*Call for Papers*

Submission deadline: June 21



In the past few years, a new evaluation methodology known as *living labs *has
been proposed as a way for researchers to be able to perform in-situ
evaluation which involve and integrate users within the research process.
The basic idea of living labs for information retrieval (IR) is that rather
than individual research groups independently developing experimental
search infrastructures and gathering their own groups of test searchers for
IR evaluations, a central and shared experimental environment is developed
to facilitate the sharing of resources in a live setting (most important of
all: users). (see ll2013.dcu.ie for further details)



Living labs would offer huge benefits to the community, such as:
availability of, potentially larger, cohorts of real users and their
behaviours, e.g. querying behaviours, for experiment purposes;
cross-comparability across research centres; and greater knowledge transfer
between industry and academia, when industry partners are involved. The
need for this methodology is further amplified by the increased reliance of
IR approaches on proprietary data; living labs are a way to bridge the data
divide between academia and industry. Progress towards realising actual
living labs has nevertheless been limited. There are many challenges to be
overcome before the benefits associated with living labs for IR can be
realised, including challenges associated with living labs architecture and
design, hosting, maintenance, security, privacy, participant recruiting,
and scenarios and tasks for use development.



This workshop aims to bring together for the first time people interested
in progressing the living labs for IR evaluation methodology. An
interactive forum for researchers to share ideas and initiate
collaborations will be provided, with the explicit goal of determining
means for progressing towards living labs for IR and formulating practical
next steps for progression.



Possible use cases in this space include, but are not limited to:



· The e-commerce domain (i.e., product search and recommendation)

· The personal search space (search of personal computer files, emails, web
pages looked at, etc)

· Medical information retrieval (e.g., patients searching for medical
information on the Internet)

· Searching Wikipedia (an open-source collection with fewer privacy
concerns than, say, personal search)



Workshop topics include, but are not limited to:



· Technical challenges:

o Designing and implementing living labs architecture

o Cost of implementation

o Maintenance and adoption

o Managing living labs infrastructure

· Evaluation challenges:

o Evaluation framework and methodology

o Evaluation measures

· Privacy and security:

o Hosting data on secure server

o Gaining subjects trust

o Coping with individuals need for privacy

o Alternates when individuals will not share their data

· Legal and ethical issues:

o User consent

o Ethics approval

o Legalities regarding release of data

o Trust between parties

o Copyright issues

o Commercial sensitivity of interaction data

· Practical challenges:

o Forming living labs for IR partners within the research community

o Obtaining commercial partners

o Alternates when commercial partners cannot be obtained

o Sharing of the physical resources

o Defining tasks and scenarios for evaluation purposes
 Poll - Should we have a Living Labs IR challenge using a medium-sized
online retailer?

As part of the Living Labs for Information Retrieval Evaluation workshop,
we are considering organising a challenge in the e-commerce domain with the
involvement of a medium-sized online retailer. The goal of this challenge
would be to (i) allow academics to work with real users and data (esp.
those who otherwise would have no access to such data) and (ii) to provide
a starting point for the discussions at the workshop.



We will set up and run this challenge if there is sufficient interest in
the community. So please let us know what you think on the poll at
http://bit.ly/LL13poll.



*Paper Submissions*

The workshop is now accepting paper submissions. Short papers (4 pages),
position papers (2 pages), and posters (2 pages) describing approaches or
ideas / challenges on the topics of the workshop are invited. Submissions
should be in ACM SIGS format. LaTeX and Word templates are available at
http://www.acm.org/sigs/publications/proceedings-templates (for LaTeX, use
the "Option 2" style).



Papers should be anonymised for double blind review and submitted in pdf
format through the EasyChair system <http://www.easychair.org/...>
https://www.easychair.org/conferences/?conf=ll2013 no later than
midnightPacific Daylight Time on June
21, 2013. Submissions will be reviewed by members of the workshop program
committee. Accepted papers will be included in the CIKM 2013 Living Labs
for Information Retrieval Evaluation Workshop proceedings, which will be
indexed in the ACM Digital Library.



*Important Dates*

*June 21, 2013**:* Deadline for paper submission (midnight Pacific Daylight
Time)

*July 22, 2013**:* Notification to authors

*August 11, 2013**:* Camera-ready paper due

*November 1, 2013**:* Workshop



*Further Information*

Further information is available on the workshop website
athttp://LL2013.dcu.ie/<http://ll2013.dcu.ie/>or by emailing the
workshop organisers.



*Workshop Organisers*

Krisztian Balog - University of Stavanger, Norway (krisztian.balog (at)
uis.no)

David Elsweiler - University of Regensburg, Germany (david (at)
elsweiler.co.uk)

Evangelos Kanoulas - Google, Switzerland (ekanoulas (at) gmail.com)

Liadh Kelly - Dublin City University, Ireland (liadh.kelly (at)
computing.dcu.ie)

Mark Smucker - University of Waterloo, Canada (msmucker (at) uwaterloo.ca)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/asis-l/attachments/20130601/9e390cb9/attachment-0001.html