[Asis-l] (extended deadline - July 27) CfP CIKM Workshop on Living Labs for IR Evaluation

[Liadh Kelly]

*Living Labs for Information Retrieval Evaluation Workshop*

http://ll2013.dcu.ie


*At CIKM 2013, **November 1 2013**, **San Francisco**, **USA*

Call for 2 & 4 page paper and demo submissions

Extended Deadline - July 27

*750€ award* - for best demo submission (see ll2013.dcu.ie for full
details)



*Call for Papers*

**
 In the past few years, a new evaluation methodology known as *living labs *has
been proposed as a way for researchers to be able to perform in-situ
evaluation which involve and integrate users within the research process.
The basic idea of living labs for information retrieval (IR) is that rather
than individual research groups independently developing experimental
search infrastructures and gathering their own groups of test searchers for
IR evaluations, a central and shared experimental environment is developed
to facilitate the sharing of resources in a live setting (most important of
all: users). (see ll2013.dcu.ie for further details)

Living labs would offer huge benefits to the community, such as:
availability of, potentially larger, cohorts of real users and their
behaviours, e.g. querying behaviours, for experiment purposes;
cross-comparability across research centres; and greater knowledge transfer
between industry and academia, when industry partners are involved. The
need for this methodology is further amplified by the increased reliance of
IR approaches on proprietary data; living labs are a way to bridge the data
divide between academia and industry. Progress towards realising actual
living labs has nevertheless been limited. There are many challenges to be
overcome before the benefits associated with living labs for IR can be
realised, including challenges associated with living labs architecture and
design, hosting, maintenance, security, privacy, participant recruiting,
and scenarios and tasks for use development.

This workshop aims to bring together for the first time people interested
in progressing the living labs for IR evaluation methodology. An
interactive forum for researchers to share ideas and initiate
collaborations will be provided, with the explicit goal of determining
means for progressing towards living labs for IR and formulating practical
next steps for progression.

Possible use cases in this space include, but are not limited to:

· The e-commerce domain (i.e., product search and recommendation)

· The personal search space (search of personal computer files, emails, web
pages looked at, etc)

· Medical information retrieval (e.g., patients searching for medical
information on the Internet)

· Searching Wikipedia (an open-source collection with fewer privacy
concerns than, say, personal search)


Workshop topics include, but are not limited to:

· Technical challenges:

o Designing and implementing living labs architecture

o Cost of implementation

o Maintenance and adoption

o Managing living labs infrastructure

· Evaluation challenges:

o Evaluation framework and methodology

o Evaluation measures

· Privacy and security:

o Hosting data on secure server

o Gaining subjects trust

o Coping with individuals need for privacy

o Alternates when individuals will not share their data

· Legal and ethical issues:

o User consent

o Ethics approval

o Legalities regarding release of data

o Trust between parties

o Copyright issues

o Commercial sensitivity of interaction data

· Practical challenges:

o Forming living labs for IR partners within the research community

o Obtaining commercial partners

o Alternates when commercial partners cannot be obtained

o Sharing of the physical resources

o Defining tasks and scenarios for evaluation purposes

*Paper & Demo Submissions*

Short papers (4 pages), position papers (2 pages), and posters (2 pages)
describing approaches or ideas / challenges on the topics of the workshop
are invited.



Demo papers (max. 4 pages) based on an implemented system that pursues one
or more aspects relevant to the interest areas of the workshop are also
invited.

Accepted submissions will be included in the CIKM 2013 Living Labs for
Information Retrieval Evaluation Workshop proceedings, which will be
indexed in the ACM Digital Library.



Further submission details on the workshop website: http://LL2013.dcu.ie/


*Important Dates*

*July 27, 2013**:* Deadline for paper submissions (midnight Pacific
Daylight Time)

*July 27, 2013:* Deadline for demo submissions (midnight Pacific Daylight
Time)



*August 6, 2013**:* Paper and demo notification to authors

*August 11, 2013**:* Camera-ready papers due

*November 1, 2013**:* Workshop


*Further Information*

Further information is available on the workshop website
athttp://LL2013.dcu.ie/<http://ll2013.dcu.ie/>or by emailing the
workshop organisers.





*Workshop Organisers*

Krisztian Balog - University of Stavanger, Norway (krisztian.balog (at)
uis.no)

David Elsweiler - University of Regensburg, Germany (david (at)
elsweiler.co.uk)

Evangelos Kanoulas - Google, Switzerland (ekanoulas (at) gmail.com)

Liadh Kelly - Dublin City University, Ireland (liadh.kelly (at)
computing.dcu.ie)

Mark Smucker - University of Waterloo, Canada (msmucker (at) uwaterloo.ca)








-- 
*Liadh Kelly, PhD*
Centre for Next Generation Localisation
Dublin City University, Ireland
email: liadh.kelly at computing.dcu.ie
web: http://www.computing.dcu.ie/~lkelly/

*Email Disclaimer: *This email and any files transmitted with it are
confidential and are intended solely for use by the addressee. Any
unauthorised dissemination, distribution or copying of this message and any
attachments is strictly prohibited. If you have received this email in
error please notify the sender and delete the message. Any views or
opinions presented in this email may solely be the views of the author and
cannot be relied upon as being those of Dublin City University. E-mail
communications such as this cannot be guaranteed to be virus free, timely,
secure or error free and Dublin City University do not accept liability for
any such matters or their consequences. Please consider the environment
before printing this Email.

*Séanadh Ríomhphoist: *Tá an ríomhphost seo agus aon chomhad a sheoltar
leis faoi rún agus is lena úsáid ag an seolaí agus sin amháin é. Tá cosc
iomlán ar scaipeadh, dháileadh nó chóipeáil neamhúdaraithe ar an
teachtaireacht seo agus ar aon cheangaltán atá ag dul leis. Má tá an
ríomhphost seo faighte agat trí dhearmad cuir sin in iúl le do thoil don
seoltóir agus scrios an teachtaireacht. D’fhéadfadh sé gurb iad tuairimí an
údair agus sin amháin atá in aon tuairimí no dearcthaí atá curtha i láthair
sa ríomhphost seo agus níor chóir glacadh leo mar thuairimí nó dhearcthaí
Ollscoil Chathair Bhaile Átha Cliath. Ní ghlactar leis go bhfuil cumarsáid
ríomhphoist den sórt seo saor ó víreas, in am, slán, nó saor ó earráid agus
ní ghlacann Ollscoil Chathair Bhaile Átha Cliath le dliteanas in aon chás
den sórt sin ná as aon iarmhairt a d’eascródh astu. Cuimhnigh ar an
timpeallacht le do thoil sula gcuireann tú an ríomhphost seo i gcló.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.asis.org/pipermail/asis-l/attachments/20130723/c76ae1cf/attachment-0001.html