[Sigia-l] Comparing search engines

[donna at maadmob.net]

I'm helping on a project that involves implementing a search engine 
and thesaurus on a big store of data (an enterprise app, not an 
informational site).

For various reasons, we are comparing two very different search 
engine/thesaurus combinations. One of the criteria for choosing a 
final product is speed, but something that I've been pushing is to also 
compare how well both combinations retrieve the 'best result' set.

So what I'm asking from you is - have you ever tried to evaluate 
whether a search engine returns a good result set. How do you figure 
out what the 'best results' are to compare???

I have been thinking of creating a large set of scenarios, working with 
users to identify what their expectations are for each scenario, and 
documenting this before running either search engine. Any better 
ideas than this?

TIA

Donna