[Sigia-l] RE: "Best Bets" the Yahoo way
jbieley
jon at bieley.com
Wed Mar 3 21:42:08 EST 2004
I think the larger issue is: maintaining a good search engine is expensive,
and where does that money come from? Right now the money is from the private
sector and so they need to recoup their investment. Advertising forms -
Banner ads, pop-up ads, intrusive Flash ads, - are all annoying. Some form
of eye-ball resale is required until companies can make money from other
schemes, like paying information producers to distributed on their
(networks) search engine results pages. Paying for inclusion, or paying for
index frequency, etc., are not unreasonable and may decrease the reliance on
advertising.
What I don't understand is why this is an issue. Commercial sites need to
make money, and as long as people dedicate personal time to crafting deep
sites, the engines will want to include them - even if hobbyist sites don't
pay for inclusion. Paid ranking is a pain but you skip over it if it is
visually distinct.
Best bets are great - but they cost money. Let's be honest, search engines
suck in comparison to good curated lists. And for all of the bottom-up
hurrah over link-based indexing, power laws of popularity (e.g., 80/20,
Paretto Principle, Zipf distribution, etc.) result in truly divergent views
being hard to find. Good, thoughtful curation by intelligent humans wins the
findability wars every time. The search engines are best at dealing with
information that moves too quickly to be curated fast enough or is too
wildly chaotic to be well organizd (e.g., operating system and hardware bug
fixes).
"The wars" started with a list of cool sites curated by two guys from
Stanford (back when commercial efforts like Pathfinder sucked). Then the Web
exploded and we needed search engines to find and index it all. Now we need
best bets to filter out the crap to find the good stuff. There are emerging
tools like Vivisimo to help organize the results. Eventually the software
and hardware will get cheap enough that there will be a good open source
search engine with distributed spidering and indexing, maybe even
distributed searching. Some university, or non-profit, or consortia will
support this and it will put downward pressure on pricing for tools from for-
profit companies.
It has already started: DMOZ (http://www.dmoz.org) and Nutch
(http://www.nutch.org/) - and of course the Internet Public Library
(http://www.ipl.org/). I didn't see a P2P spider/indexer/retrieval engine
but check out Spidering Hacks from O'Reilly and let's start one. I'll even
provide a name: Cristo after the artist who wraps objects. Let's wrap the
Web in a P2P search engine.
Do I hate the commercialization? Sure, I don't want the world to look like
Blade Runner. But I love the tools that market power has been able to get
into our hands. Let's build new tools to replace what we don't like.
Jonathan Bieley
MapleStar Consulting
More information about the Sigia-l
mailing list