[Sigia-l] RE: "Best Bets" the Yahoo way

[jbieley]

I think the larger issue is: maintaining a good search engine is expensive, 
and where does that money come from? Right now the money is from the private 
sector and so they need to recoup their investment. Advertising forms -  
Banner ads, pop-up ads, intrusive Flash ads, - are all annoying. Some form 
of eye-ball resale is required until companies can make money from other 
schemes, like paying information producers to distributed on their 
(networks) search engine results pages. Paying for inclusion, or paying for 
index frequency, etc., are not unreasonable and may decrease the reliance on 
advertising. 

What I don't understand is why this is an issue. Commercial sites need to 
make money, and as long as people dedicate personal time to crafting deep 
sites, the engines will want to include them - even if hobbyist sites don't 
pay for inclusion. Paid ranking is a pain but you skip over it if it is 
visually distinct. 

Best bets are great - but they cost money. Let's be honest, search engines 
suck in comparison to good curated lists. And for all of the bottom-up 
hurrah over link-based indexing, power laws of popularity (e.g., 80/20, 
Paretto Principle, Zipf distribution, etc.) result in truly divergent views 
being hard to find. Good, thoughtful curation by intelligent humans wins the 
findability wars every time. The search engines are best at dealing with 
information that moves too quickly to be curated fast enough or is too 
wildly chaotic to be well organizd (e.g., operating system and hardware bug 
fixes).

"The wars" started with a list of cool sites curated by two guys from 
Stanford (back when commercial efforts like Pathfinder sucked). Then the Web 
exploded and we needed search engines to find and index it all. Now we need 
best bets to filter out the crap to find the good stuff. There are emerging 
tools like Vivisimo to help organize the results. Eventually the software 
and hardware will get cheap enough that there will be a good open source 
search engine with distributed spidering and indexing, maybe even 
distributed searching. Some university, or non-profit, or consortia will 
support this and it will put downward pressure on pricing for tools from for-
profit companies.

It has already started: DMOZ (http://www.dmoz.org) and Nutch 
(http://www.nutch.org/) - and of course the Internet Public Library 
(http://www.ipl.org/). I didn't see a P2P spider/indexer/retrieval engine 
but check out Spidering Hacks from O'Reilly and let's start one. I'll even 
provide a name: Cristo after the artist who wraps objects. Let's wrap the 
Web in a P2P search engine. 

Do I hate the commercialization? Sure, I don't want the world to look like 
Blade Runner. But I love the tools that market power has been able to get 
into our hands. Let's build new tools to replace what we don't like.


Jonathan Bieley
MapleStar Consulting