[Asis-l] CfP: CIKM 2nd Living Labs for Information Retrieval Evaluation Workshop

Wed May 7 09:07:10 EDT 2014

------------------------------------------------------------------------------

The 2nd Living Labs for Information Retrieval Evaluation Workshop @ CIKM
2014

7 November, 2014, Shanghai, China
http://living-labs.net/ll14/

Call for Papers & Demos - Submission deadline: July 30
Challenge Track - Submission deadline: August 20

https://groups.google.com/forum/#!forum/living-labs

--------------------------------------------------------------------------------

In the past few years, a new evaluation methodology known as living labs
has been proposed as a way for researchers to be able to perform in-situ
evaluation which involve and integrate users within the research process.
The basic idea of living labs for information retrieval (IR) is that rather
than individual research groups independently developing experimental
search infrastructures and gathering their own groups of test searchers for
IR evaluations, a central and shared experimental environment is developed
to facilitate the sharing of resources in a live setting (most important of
all: users).

Living labs would offer huge benefits to the community, such as:
availability of, potentially larger, cohorts of real users and their
behaviours, e.g. querying behaviours, for experiment purposes;
cross-comparability across research centres; and greater knowledge transfer
between industry and academia, when industry partners are involved. The
need for this methodology is further amplified by the increased reliance of
IR approaches on proprietary data; living labs are a way to bridge the data
divide between academia and industry. Progress towards realising actual
living labs has nevertheless been limited. There are many challenges to be
overcome before the benefits associated with living labs for IR can be
realised, including challenges associated with living labs architecture and
design, hosting, maintenance, security, privacy, participant recruiting,
and scenarios and tasks for use development.

The 1st Living Labs for Information Retrieval Evaluation (LL’13) workshop
at CIKM 2013 was a first attempt to bring people, both from academia and
industry, together to discuss challenges and to formulate practical next
steps. The workshop was successful in identifying and documenting possible
further directions. The goal of the second edition of the workshop is to
continue our community building efforts around living labs for IR and to
pursue the directions set out at LL’13. As part of this, a challenge with
shared tasks in the e-commerce space and local domain search is planned.

Possible use cases in this space include, but are not limited to:

-The e-commerce domain (i.e., product search and recommendation)
-Local domain search (e.g., university search engines)
-The personal search space (search of personal computer files, emails, web
pages looked at, etc.)
-Medical information retrieval (e.g., patients searching for medical
information on the Internet)
-Searching Wikipedia (an open-source collection with fewer privacy concerns
than, say, personal search)

Workshop topics include, but are not limited to:

Privacy and security:
-Hosting data on secure server
-Gaining subjects trust
-Coping with individuals need for privacy
-Alternates when individuals will not share their data

Legal and ethical issues:
-User consent
-Ethics approval
-Legalities regarding release of data
-Trust between parties
-Copyright issues
-Commercial sensitivity of interaction data

Technical challenges:
-Designing and implementing living labs architecture
-Cost of implementation
-Maintenance and adoption
-Managing living labs infrastructure

Practical challenges:
-Forming living labs for IR partners within the research community
-Obtaining commercial partners
-Alternates when commercial partners cannot be obtained
-Defining tasks and scenarios for evaluation purposes

-----------------
Paper Submissions
-----------------
The workshop is now accepting paper submissions. Short papers (4 pages),
position papers (2 pages), and posters (2 pages) describing approaches or
ideas / challenges on the topics of the workshop are invited. Submissions
should be in ACM SIGS format. LaTeX and Word templates are available at
http://www.acm.org/sigs/publications/proceedings-templates (for LaTeX, use
the “Option 2″ style).

Papers should be anonymised for double blind review and submitted in PDF
format through the EasyChair system (
https://www.easychair.org/conferences/?conf=ll2014) no later than midnight
Pacific Daylight Time on July 30, 2014. Submissions will be reviewed by
members of the workshop program committee. Accepted papers will be included
in the CIKM 2014 Living Labs for Information Retrieval Evaluation Workshop
proceedings, which will be indexed in the ACM Digital Library.

----------------
Demo Submissions
----------------
We invite researchers and practitioners to present their innovative
prototypes or practical developments in a dedicated demo track at the
workshop. Demo submissions must be based on an implemented system that
pursues one or more aspects relevant to the interest areas of the workshop.

Authors are strongly encouraged to target scenarios that are rooted in
real-world applications. One way to think about this is by considering the
following: as a company operating a website/service/application, what
methods could allow various academic groups to experiment with specific
components of this website/service/application?

In particular, we seek prototypes that define specific component(s) in the
context of some website/service/application, and allow for the testing and
evaluation of alternative methods for that component. One example is search
within a specific vertical (such as product or travel search engine), but
we encourage authors to think outside the (search) box.

Demonstration papers (max. 4 pages) must be submitted electronically
through the EasyChair system (
https://www.easychair.org/conferences/?conf=ll2014), in PDF format and
formatted using the ACM SIGS format (LaTeX and Word templates are available
at http://www.acm.org/sigs/publications/proceedings-templates; for LaTeX,
use the “Option 2″ style). Each submission should either include a URL for
the web site of the system, or a URL of a shared code repository, or a URL
to a 2-3 minute video demonstrating the system. The review process for
demonstration submission is *not* double blind. Submissions should be made
no later than midnight Pacific Daylight Time on July 30, 2014. Submissions
will be reviewed by members of the workshop program committee. Accepted
demo papers will be included in the CIKM 2014 Living Labs for Information
Retrieval Evaluation Workshop proceedings, which will be indexed in the ACM
Digital Library.

---------------------------
Challenge Track Submissions
---------------------------
The workshop also features a living labs challenge. The challenge focuses
on two use-cases: product search (on an e-commerce site) and local domain
search (on a university’s website). The challenge will run from June 15 –
August 15. During the first month of the challenge participants will have
the opportunity to tune their system through the product and local domain
search websites. The second month of the challenge will examine the
performance of participants’ systems on these search websites. Further
details are available at: http://living-labs.net/challenge/.

Challenge papers (max. 4 pages) should describe the participant’s system
design approach and the results they obtained in the living labs challenge.
Papers must be submitted electronically through the EasyChair system (
https://www.easychair.org/conferences/?conf=ll2014), in PDF format and
formatted using the ACM SIGS format (LaTeX and Word templates are available
at http://www.acm.org/sigs/publications/proceedings-templates; for LaTeX,
use the “Option 2″ style). The review process for challenge submissions is
*not* double blind. Submissions should be made no later than midnight
Pacific Daylight Time on August 20, 2014. Submissions will be reviewed by
members of the workshop program committee. Accepted challenge papers will
be included in the CIKM 2014 Living Labs for Information Retrieval
Evaluation Workshop proceedings, which will be indexed in the ACM Digital
Library.

---------------
Important Dates
---------------
July 30, 2014 - Paper & Demo Track submission deadline (midnight Pacific
Daylight Time)

August 20, 2014 – Individual challenge papers due (midnight Pacific
Daylight Time) [Challenge runs June 15 – August 15]

August 30, 2014 – Notification to authors

September 10, 2014 – Camera-ready papers due

November 7, 2014 - Workshop

-------------------
Further Information
-------------------
Further information is available on the workshop website (
http://living-labs.net/ll14/) or by emailing the workshop organisers.

-------------------
Workshop Organisers
-------------------
Krisztian Balog – University of Stavanger, Norway (krisztian.balog (at)
uis.no)
Liadh Kelly – Dublin City University, Ireland (liadh.kelly (at)
computing.dcu.ie)
Anne Schuth – University of Amsterdam, The Netherlands (anne.schuth (at)
uva.nl)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/asis-l/attachments/20140507/95a87dbe/attachment.html>