[Sigia-l] Can you recommend site mapping program?; also look for words of wisdom

[Eric Scheid]

In addition to advice to send in the bots to crawl your site, I would
recommend you get your mitts on the traffic logs and find out what pages are
being requested. Then get a bot to ping each of those URLs to see if they
still exist (if they ever did). (I'd recommend you manually scan and purge
obviously bad URLs first though).

Sometimes a site has sections which are not normally accessible via an
automated crawl (eg. needs a click on a flash widget).

Sometimes old pages have been removed from the navigation of other pages,
but were left on the server ... and they still get traffic (mostly from
search engines that saw the pages when they were originally linked in, but
also from bookmarks and also links from 3rd party sites).

e.