[Asis-l] CfP & IR Challenge Open: CIKM 2nd Living Labs for Information Retrieval Evaluation Workshop
Liadh Kelly
lkelly at computing.dcu.ie
Tue Jul 8 07:42:39 EDT 2014
===================================================================================
The 2nd Living Labs for Information Retrieval Evaluation Workshop @ CIKM
2014
7 November, 2014, Shanghai, China
http://living-labs.net/ll14/
Call for Papers & Demos - Submission deadline: July 30
Challenge Track - Submission deadline: August 20
https://groups.google.com/forum/#!forum/living-labs
===================================================================================
** IR CHALLENGE NOW OPEN **
Offering access to product/webpage information, usage and query log data
through an API and facilities to test approaches in a live setting.
Register at: http://living-labs.net/challenge/
===================================================================================
In the past few years, a new evaluation methodology known as living labs
has been proposed as a way for researchers to be able to perform in-situ
evaluation which involve and integrate users within the research
process. The basic idea of living labs for information retrieval (IR) is
that rather than individual research groups independently developing
experimental search infrastructures and gathering their own groups of
test searchers for IR evaluations, a central and shared experimental
environment is developed to facilitate the sharing of resources in a
live setting (most important of all: users).
Living labs would offer huge benefits to the community, such as:
availability of, potentially larger, cohorts of real users and their
behaviours, e.g. querying behaviours, for experiment purposes;
cross-comparability across research centres; and greater knowledge
transfer between industry and academia, when industry partners are
involved. The need for this methodology is further amplified by the
increased reliance of IR approaches on proprietary data; living labs are
a way to bridge the data divide between academia and industry. Progress
towards realising actual living labs has nevertheless been limited.
There are many challenges to be overcome before the benefits associated
with living labs for IR can be realised, including challenges associated
with living labs architecture and design, hosting, maintenance,
security, privacy, participant recruiting, and scenarios and tasks for
use development.
The 1st Living Labs for Information Retrieval Evaluation (LL’13)
workshop at CIKM 2013 was a first attempt to bring people, both from
academia and industry, together to discuss challenges and to formulate
practical next steps. The workshop was successful in identifying and
documenting possible further directions. The goal of the second edition
of the workshop is to continue our community building efforts around
living labs for IR and to pursue the directions set out at LL’13. As
part of this, a live challenge with shared tasks in the e-commerce space
and local domain search is now running, offering access to
product/webpage information, usage and query log data through an API and
facilities to test approaches in a live setting (register at
http://living-labs.net/challenge/ ).
Possible use cases in this space include, but are not limited to:
-The e-commerce domain (i.e., product search and recommendation)
-Local domain search (e.g., university search engines)
-The personal search space (search of personal computer files, emails,
web pages looked at, etc.)
-Medical information retrieval (e.g., patients searching for medical
information on the Internet)
-Searching Wikipedia (an open-source collection with fewer privacy
concerns than, say, personal search)
Workshop topics include, but are not limited to:
Privacy and security:
-Hosting data on secure server
-Gaining subjects trust
-Coping with individuals need for privacy
-Alternates when individuals will not share their data
Legal and ethical issues:
-User consent
-Ethics approval
-Legalities regarding release of data
-Trust between parties
-Copyright issues
-Commercial sensitivity of interaction data
Technical challenges:
-Designing and implementing living labs architecture
-Cost of implementation
-Maintenance and adoption
-Managing living labs infrastructure
Practical challenges:
-Forming living labs for IR partners within the research community
-Obtaining commercial partners
-Alternates when commercial partners cannot be obtained
-Defining tasks and scenarios for evaluation purposes
-----------------
Paper Submissions
-----------------
The workshop is now accepting paper submissions. Short papers (4 pages),
position papers (2 pages), and posters (2 pages) describing approaches
or ideas / challenges on the topics of the workshop are invited.
Submissions should be in ACM SIGS format. LaTeX and Word templates are
available at http://www.acm.org/sigs/publications/proceedings-templates
(for LaTeX, use the “Option 2″ style).
Papers should be anonymised for double blind review and submitted in PDF
format through the EasyChair system
(https://www.easychair.org/conferences/?conf=ll2014) no later than
midnight Pacific Daylight Time on July 30, 2014. Submissions will be
reviewed by members of the workshop program committee. Accepted papers
will be included in the CIKM 2014 Living Labs for Information Retrieval
Evaluation Workshop proceedings, which will be indexed in the ACM
Digital Library.
----------------
Demo Submissions
----------------
We invite researchers and practitioners to present their innovative
prototypes or practical developments in a dedicated demo track at the
workshop. Demo submissions must be based on an implemented system that
pursues one or more aspects relevant to the interest areas of the workshop.
Authors are strongly encouraged to target scenarios that are rooted in
real-world applications. One way to think about this is by considering
the following: as a company operating a website/service/application,
what methods could allow various academic groups to experiment with
specific components of this website/service/application?
In particular, we seek prototypes that define specific component(s) in
the context of some website/service/application, and allow for the
testing and evaluation of alternative methods for that component. One
example is search within a specific vertical (such as product or travel
search engine), but we encourage authors to think outside the (search) box.
Demonstration papers (max. 4 pages) must be submitted electronically
through the EasyChair system
(https://www.easychair.org/conferences/?conf=ll2014), in PDF format and
formatted using the ACM SIGS format (LaTeX and Word templates are
available at http://www.acm.org/sigs/publications/proceedings-templates;
for LaTeX, use the “Option 2″ style). Each submission should either
include a URL for the web site of the system, or a URL of a shared code
repository, or a URL to a 2-3 minute video demonstrating the system. The
review process for demonstration submission is *not* double blind.
Submissions should be made no later than midnight Pacific Daylight Time
on July 30, 2014. Submissions will be reviewed by members of the
workshop program committee. Accepted demo papers will be included in the
CIKM 2014 Living Labs for Information Retrieval Evaluation Workshop
proceedings, which will be indexed in the ACM Digital Library.
---------------------------
Challenge Track Submissions
---------------------------
The workshop also features a living labs challenge. The challenge
focuses on two use-cases: product search (on an e-commerce site) and
local domain search (on a university’s website). For each of these
tasks, challenge participants have access to product/webpage
information, usage and query log data through an API and are able to
test their approaches in a live setting.
The challenge is now open and runs until August 15. During the first
month of the challenge participants will have the opportunity to tune
their system through the product and local domain search websites. The
second month of the challenge will examine the performance of
participants’ systems on these search websites. Further details are
available at: http://living-labs.net/challenge/.
Challenge papers (max. 4 pages) should describe the participant’s system
design approach and the results they obtained in the living labs
challenge. Papers must be submitted electronically through the EasyChair
system (https://www.easychair.org/conferences/?conf=ll2014), in PDF
format and formatted using the ACM SIGS format (LaTeX and Word templates
are available at
http://www.acm.org/sigs/publications/proceedings-templates; for LaTeX,
use the “Option 2″ style). The review process for challenge submissions
is *not* double blind. Submissions should be made no later than midnight
Pacific Daylight Time on August 20, 2014. Submissions will be reviewed
by members of the workshop program committee. Accepted challenge papers
will be included in the CIKM 2014 Living Labs for Information Retrieval
Evaluation Workshop proceedings, which will be indexed in the ACM
Digital Library.
---------------
Important Dates
---------------
July 30, 2014 - Paper & Demo Track submission deadline (midnight Pacific
Daylight Time)
August 20, 2014 – Individual challenge papers due (midnight Pacific
Daylight Time) [Challenge runs from now – August 15]
August 30, 2014 – Notification to authors
September 10, 2014 – Camera-ready papers due
November 7, 2014 - Workshop
-------------------
Further Information
-------------------
Further information is available on the workshop website
(http://living-labs.net/ll14/) or by emailing the workshop organisers.
-------------------
Workshop Organisers
-------------------
Krisztian Balog – University of Stavanger, Norway (krisztian.balog (at)
uis.no)
Liadh Kelly – Dublin City University, Ireland (liadh.kelly (at)
computing.dcu.ie)
Anne Schuth – University of Amsterdam, The Netherlands (anne.schuth (at)
uva.nl)
More information about the Asis-l
mailing list