[Sigia-l] Comparing search engines

[david landry]

What you seem to be asking about is what I would call "relevance" of 
results.  There's a lot of research on how to measure relevance 
quantitatively, typically, by measuring "recall" (completeness of the 
result set) and "precision" (amount of noise in the result set), which 
are inversely related.  The TREC conference is devoted entirely to this 
topic is probably worth a look (http://trec.nist.gov/).

The approach you outline however is perfectly valid.  Identify a corpus 
of documents.  Define a set of queries.  Determine from a set of users 
what results they would expect to see for each query.  Benchmark 
against the search engine.  Realize though that each user's 
determination of what constitutes a relevant result will be somewhat 
unique.

david landry


----- Original Message -----
From: donna at maadmob.net
Date: Thursday, June 19, 2003 4:38 am
Subject: [Sigia-l] Comparing search engines

> I'm helping on a project that involves implementing a search 
> engine 
> and thesaurus on a big store of data (an enterprise app, not an 
> informational site).
> 
> For various reasons, we are comparing two very different search 
> engine/thesaurus combinations. One of the criteria for choosing a 
> final product is speed, but something that I've been pushing is to 
> also 
> compare how well both combinations retrieve the 'best result' set.
> 
> So what I'm asking from you is - have you ever tried to evaluate 
> whether a search engine returns a good result set. How do you 
> figure 
> out what the 'best results' are to compare???
> 
> I have been thinking of creating a large set of scenarios, working 
> with 
> users to identify what their expectations are for each scenario, 
> and 
> documenting this before running either search engine. Any better 
> ideas than this?
> 
> TIA
> 
> Donna
> 
> ------------
> When replying, please *trim your post* as much as possible.
> *Plain text, please; NO Attachments
> 
> Searchable list archive:   http://www.info-arch.org/lists/sigia-l/
> ________________________________________
> Sigia-l mailing list -- post to: Sigia-l at asis.org
> Changes to subscription: 
> http://mail.asis.org/mailman/listinfo/sigia-l
>