[Asis-l] Announcing MIREX Grand Challenge 2015: User Experience Task

[Downie, J Stephen]

Dear Colleagues:

We are pleased to announce that the Music Information Retrieval Evaluation eXchange (MIREX) Grand Challenge 2015 User Experience (GC15UX) can now begin. In response to the feedback received at ISMIR 2014, we have created a new dataset and slightly changed the evaluation criteria for this year's event.

GC15UX is a holistic, user-centered evaluation of the user experience in interacting with complete, user-facing music information retrieval (MIR) systems. This aims at inspiring the development of complete MIR systems and promoting the notion of user experience as a first-class research objective in the MIR community. Please share this announcement with other interested individuals and groups who might be new to the world of MIR.

Each team will develop one complete MIR system that can play songs from the Jamendo collection. Later, evaluators will evaluate participating systems while assuming a defined yet open task. Finally, the first round of results will be released in time for MIREX in Spain as part of ISMIR 2015. Specific dates and locations will be updated on the wiki once they are determined.

The 10,000 tracks in GC15UX are sampled (w.r.t. maximizing music variety) from the Jamendo collection with CC-BY license and made available for participants (system developers) to download to build their systems. It represents a randomly chosen subset the content available at Jamendo that is published under the terms of the Creative Commons Attribution-Non-Commercial-ShareAlike (by-nc-sa), where user-supplied data has tagged a track with 1 or more genre categories. Importantly, we have created a new dataset, which has better distribution of genres and artists in response to feedback from last year's evaluation. For more details about usage of this dataset, see the LICENSE.txt file contained in the downloaded files.

The dataset contains the MP3 tracks and the metadata the Jamendo site publishes on the respective items (represented in JSON format), retrieved using the site's API (6th July 2015). The dataset is available both zipped up and as a tar-ball (you only need one of these); however, at 60+ Gb it is a non-trival size of file to download over the web, and so we suggest you install a Download Manager extension to your browser if you do not already have one and make use of that. In a test using the DownThemAll! extension to Firefox, downloading the dataset between University of Illinois at Urbana-Champaign and Waikato University in New Zealand took a little under 2 hours.

You need to register to download the main dataset. If you have already submitted something to MIREX, your previous registration should still work. Please note that you need to login as a submitter. Once logged in as a submitter, you can find a download link on the left.

This year GC15UX adopts the two-phase model with two evaluations. The first phase will end by the ISMIR conference and we will disclose preliminary results at the conference like what we did last year. Then, phase II will start. Participating developers can continue improving their systems based on the feedback from the first phase and another round of evaluation will be conducted in February. We believe that this model serves the developers well since it is in accordance with the iterative nature of user-centered design. In this way, the developers will also have enough time to develop their complete MIR systems. 

Lastly, there is more to come. We are now working on creating another special Grand Challenge. We hope that we can announce the new GC soon.     

Information about GC14UX:
http://www.music-ir.org/mirex/wiki/2015:GC15UX 

Data available after logging in:
http://www.music-ir.org/mirex/gc15ux/ 

Informetric profile of the selected tracks:
http://www.music-ir.org/mirex/wiki/2015:GC15UX:JSON_Metadata 

Submission deadlines :
phase I:  21 September 2015
phase II: 21 February 2016 

Here are our email addresses in case you have any questions or comments. Please feel free to contact either of us. 
Prof. Downie: jdownie at illinois.edu  
Kahyun Choi:  ckahyu2 at illinois.edu

********************************************************** 
   "Research funding makes the world a better place"
********************************************************** 
J. Stephen Downie, PhD     
Associate Dean for Research     
Professor     
Graduate School of Library and Information Science    
University of Illinois at Urbana-Champaign    
[Vox/Voicemail] (217) 649-3839