[Sigia-l] Don't submit websites to search engines?

[Alex Wright]

As the author of the Deep Web article, I just want to respond to a couple of
Ziya's points:

>> Those of us who place our faith in the Googlebot may be surprised to
>> learn that the big search engines crawl less than 1 percent of the 
>> known Web.

> Is there any concrete evidence of this?

Yes. In February, Google announced plans to expand its searchable index to
about 6 billion documents; reputable sources like Yahoo! and Brightplanet
have pegged the size of the Deep Web at upwards of 100 billion documents.

> Cut and paste "The Chemical and Biological Warfare Threat" into Google,
> elapsed time less than 0.28 seconds:
>
> http://www.sci.sdsu.edu/classes/biology/bio610/bernstein/PDFS/
> Dr.Sabbadini/warfare.pdf

The document Ziya refers to is not the CIA report I mentioned in my article;
it just happens to share the same title. This article was written by Dr.
Roger Sabbadini of the Institute for International Security and Conflict
Resolution at San Diego State University. The CIA report still does not
appear in Google (or Yahoo!) search results.

regards,
alex
-----------
alex wright
alex at agwright.com | www.agwright.com