[Siguse-l] CfP: TPDL Workshop User-Oriented Evaluation of Digital Library Interfaces

[Nils Pharo]

Call for Papers and Call for Participation

=====================================================================
User-Oriented Evaluation of Digital Library Interfaces

(in conjunction with TPDL 2011 -
Intl. Conf. Theory and Practice of Digital Libraries 2011)

Berlin, Germany, September 29, 2011

http://www.is.inf.uni-due.de/misc/tpdl_dlui.html
=====================================================================

User-oriented evaluation of methods and systems in the area of digital
libraries is often restricted to small numbers of users, or uses only
shallow observation data like system logs or questionnaires. This
workshop aims at carrying out a large collaborative evaluation prior to
the workshop, where the collected data will be shared among all
participants. At the meeting itself, we will discuss more generally the
challenges of evaluating interfaces through this kind of experiments,
with the experimental data used to exemplify an evaluation method.

The workshop will be part of the 2011 International Conference on Theory
and Practice of Digital Libraries (formerly ECDL) in Berlin, Germany
(http://www.tpdl2011.org). The format of the workshop will be as follows:
1. Presentation and discussion of results of the joint evaluation
2. Presentations of the workshop participants about their own work
in this area, as well as position statements
3. Discussion of plans for future activities of this kind.

The initiators of this workshop have several years of experience in
organizing large collaborative evaluations, namely the INEX interactive
track, each involving 50-100 users. With the planned workshop, the goal
is to bring this activity closer to the digital library community, by
using a digital library test collection that consists of the Amazon book
catalog and corresponding LibraryThing data, thus comprising also social
data such as user reviews, tags and recommendations. By studying user
search behavior with this collection, a large number of interactions can
be collected as a basis for the user interface comparison, but also
enrich the existing social media corpus can be enriched with interaction
data, that enable researchers to understand how users exploit social
annotations like reviews and tags, as well as evaluate algorithms for
tagging and recommendation (through prediction).

Research teams participating in the evaluation effort will have access
to all the experimental data collected, thus allowing them to carry out
their own analyses. The experimental system will be provided by the
University of Duisburg-Essen, Germany, consisting of two different user
interfaces for the ezDL framework (http://www.ezdl.de), which also
provides various logging methods as well as tools for carrying out user
experiments. For the former, the system not only logs user interactions,
but also provides an object-level interface for collecting eye-tracking
data. The system also includes a questionnaire tool for collecting
users' answers, and the experiment scheduler presents tasks in a
predefined order and keeps track of the time spent on each task. Since
the front-end is based on Java Webstart, it can be easily installed on
any machine supporting Java, while the back-end runs at a central site.

For the joint evaluation, workshop participants are encouraged to
recruit at least 8 test subjects that have to perform three tasks with
one of the systems (taking at most 90 minutes per person).


Evaluation participation:
------------------------------
For participating in this initiative, please send an email to
norbert.fuhr at uni-due.de immediately, but no later than May 30, 2011.


Workshop submissions:
------------------------------
As workshop contributions, we seek reports and position statements on
topics such as
* Innovative user interfaces for DL
* Evaluations of specific DL interfaces
* Evaluation methodology
* Evaluation infrastructure: testbeds, frameworks, ...

Workshop submissions should be uploaded to our EasyChair web site at
https://www.easychair.org/conferences/?conf=tpdlui2011


Important Dates:
---------------------
June 15: Deadline for paper submissions
July 15: Notification of authors.
May 30: Deadline for sign-up for evaluation participation
May-June: evaluation sessions
July 15: collected data available for all participants of the evaluations
September 29: Workshop in Berlin, Germany


Program committee:
-------------------------
Norbert Fuhr, U. of Duisburg-Essen, Germany
Birger Larsen, Royal School of Library and Information Science, Denmark
Ragnar Nordlie, Oslo University College, Norway
Arjen de Vries, Delft University, Netherlands


-- 
      Prof. Dr. Norbert Fuhr     Street Address:
      Information Engineering     Lotharstr. 65
Department of Computer Science  47057 Duisburg
   and Applied Cognitive Science  fon:  +49 203 379-2524, -3556
University of Duisburg-Essen    fax:  +49 203 379-2549
        47048 Duisburg            mail:norbert.fuhr at uni-due.de
            Germany               Web:www.is.inf.uni-due.de

-- 
      Prof. Dr. Norbert Fuhr     Street Address:
      Information Engineering     Lotharstr. 65
Department of Computer Science  47057 Duisburg
   and Applied Cognitive Science  fon:  +49 203 379-2524, -3556
University of Duisburg-Essen    fax:  +49 203 379-2549
        47048 Duisburg            mail:norbert.fuhr at uni-due.de
            Germany               Web:www.is.inf.uni-due.de

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Attached Message Part
Url: http://mail.asis.org/pipermail/siguse-l/attachments/20110407/757aecac/attachment.pl