[Sigia-l] Natural Language Searching vs Keyword Searching

[Gary Carlson]

In my limited experience I came to the following, subjective,
conclusions.  Natural Language processing (along with Baysian, vector
anaylsis, etc) does do a fair to middlin job on large doc sets, and may
be the best solution when you have a large corpus of untagged content
which is not mission critical.  

Tagging documents can be very expensive, so you need a good business
justification to do so.  Even then, human tagging is not necessarily
consistent or reliable.  Also, keyword tagging by humans becomes less
effective unless there is a consistent and controlled list of terms that
is used and understood throughout an organization.  This is also an
expensive endeavor.

As for the claims by the company you mentioned.  I have seen some
products that do a decent job, however, they should be able to demo
against your doc set before you buy.  As for the their claim that no
metadata will be necessary... Well, that really depends upon the
business requirements you have.  If you simply need to allow users to
search against in coming news feeds or all the docs on a fileshare,
maybe you do not need to do a lot with metadata.  However, if your
requirement is that you be able to search for the definitive sales
reports by region, or all documents related to product X, then I think
metadata would be an important part of your solution.

Good Luck

-----Original Message-----
From: sigia-l-admin at asis.org [mailto:sigia-l-admin at asis.org] On Behalf
Of david_fiorito at vanguard.com
Sent: Friday, February 20, 2004 11:25 AM
To: sigia-l at asis.org; sigia-l-admin at asis.org
Subject: RE: [Sigia-l] Natural Language Searching vs Keyword Searching

The company pitching natural language search to us talks a lot about the

ability to return an answer to a question rather than a set of links to 
pages like a classic keyword search.  I for one am very skeptical of
this 
kind of claim.  I am especially skeptical since they are trying to sell
us 
on the idea that we will never need metadata again since their software 
can somehow "understand" the content of a page.  The whole thing smells
a 
little fishy to me - thus the reason for my original post.

Cheers,

Dave






"Ken Bryson" <kbryson at alias.com>
Sent by: sigia-l-admin at asis.org
02/20/2004 12:39 PM

 
        To:     <sigia-l at asis.org>
        cc:     (bcc: David Fiorito/IT/VGI)
        Subject:        RE: [Sigia-l] Natural Language Searching vs
Keyword Searching







>> Does anyone have any thoughts on the relative merits of these two
types
of search?  In >>what context is one more effective that the other?
Pros/Cons of each?  Thoughts?
>>Opinions?

> But if a natural language search returns several hundred or several
thousands of
> documents, then the search is pretty useless. That's when keyword 
tagging
becomes
> important.


I don't think it's just a matter of natural language sucks in all 
instances
over 10 documents. There's no reason you can't make good use of metadata

and
thesauri AND put a natural language front end search on it.  When you do
that, the differences between natural language and keyword boolean 
searching
becomes clear, at least from my own research.

Boolean searches are better for known item searches, and natural
language
searches are *sometimes* better for subject searches.  It all really 
depends
on the skill, or lack thereof, of the searcher.


-kb

------------
When replying, please *trim your post* as much as possible.
*Plain text, please; NO Attachments

Searchable list archive:   http://www.info-arch.org/lists/sigia-l/
________________________________________
Sigia-l mailing list -- post to: Sigia-l at asis.org
Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l



------------
When replying, please *trim your post* as much as possible.
*Plain text, please; NO Attachments

Searchable list archive:   http://www.info-arch.org/lists/sigia-l/
________________________________________
Sigia-l mailing list -- post to: Sigia-l at asis.org
Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l