[Sigia-l] RE: "Best Bets" the Yahoo way

[Avi Rappoport]

There are several cases where web sites need to actually interact 
with the indexer to get content into a search engine.  A paid 
inclusion program is one solution.

For example, content on a subscription site such as the New York 
Times archives would be very useful for searching.  I'd often be 
willing to pay for the article, if I could just *find* it.  If the 
spammers hadn't abused cloaking and sniffing, it would be easy enough 
to write a special version for search indexers to access.  As it is, 
a trusted source inclusion plan makes a lot of sense, even if 
companies pay for it.

Another case is dynamic pages generated in response to forms.  If 
you're really smart, like Amazon, you present these to search 
indexers in the form of static pages, and provide links as well as 
forms.  Most companies can't get their brains around this concept. 
For them, it's easier to just send an XML feed to a search engine.

Finally, if some of your content changes frequently but 
non-deterministically, the search engine spiders will often fail to 
find the changes, leaving stale content in the index.  That does harm 
to the end-user, the search quality and the content provider.

I worry about pay for inclusion, and I think it's worth keeping an 
eye on, but I'm not convinced it's by nature a bad thing.  Most of 
these issues can be addressed by Search Engine Optimizers, and they 
sometimes stray into spamming or miss a policy change.  There are 
advantages to removing the third-party between site publishers and 
search engines.

Avi

-- 
   Avi Rappoport, Search Engine Consultant <mailto:avirr at searchtools.com>
   Complete Guide to Search Engines for Web Sites and Intranets
             <http://www.searchtools.com>