[Sigia-l] how are you making wireframes for Flash projects?

[Karl Fast]

> Google attempts to defeat this workaround, by retrying pages using a
> non-spider UA string 

Last fall I wrote a spider to automatically query Altavista,
AlltheWeb, and Google and parse the results. It would send a query
and then screenscape the resulting HTML for the linkcount figure.

Google detected this and returned a special page saying, in effect,
"you can't run a spider against us because it's against our usage
policy, but you can use the Google API." Which I did. Altavista and
AlltheWeb didn't prevent me from doing this.

Now my script was simple. The Google query was just a few lines of
python using the standard library. You could be more sophisticated
in how you emulate a 'real' browser, but Google does automatically
detect the simple case. I never bothered to figure out exactly how
it was doing this.


--karl
http://www.livingskies.com/