[Sigia-l] Site map generator?

[Lyle Kantrovich]

On 12/3/05, Listera <listera at rcn.com> wrote:
> The paragraph you isolated refers to what the tools we've had for at least a
> decade *do*. Are you disputing that we have tools that do the things I
> cited?

No, I'm saying that the things you cited don't help in this situation.
 The original question from Jan (the person we're trying to help here)
was

"Does anyone know of a tool or application where you can take a fully
constructed website and generate a simple site map from it without
having to go through the rigmarole of having to create it by hand by
going through the site?"

Jan still doesn't have an answer, even though you mention pre-existing
tools that "discover links, parse out relationships, and finally
cluster such relationships"...she still doesn't have a recommendation
for an existing tool to do what she needs.

It's analogous to someone asking for a tack hammer or a ball pein
hammer, and having someone state that "we've had sledge hammers and
jack hammers around for years."
(see http://www.hammersource.com/Tack_Hammers.html)

When I said "sometimes logical theories don't work out in the real
world", I meant that, while in theory you might be able to drive tacks
with a 10 pound sledge hammer, or shape sheet metal with a jack
hammer, those tools aren't well-suited to those jobs.

> > I've tried many a "site mapper" in my day, and found a few that could
> > work for very small sites...but nothing that scaled well.
>
> Regressive relationship discovery apps in various forms have been around
> probably longer than two decades. Having built one 11 years ago as part of a
> tagging engine, I can tell you that such software can scale deeper and wider
> than any human can by an order of magnitude.

The reason Jan's looking for a tool (and why I have looked for one in
the past) IS because software scales well.

> > 1. Directory and file structure doesn't always map to the semantic or
> > content structure
>
> So? Use other criteria. A site *discovery* tool doesn't just have to map
> file directories.

My point was that existing tools tend to use either directory/file
structure or link structure - neither of which works well.


> As you say, there are issues with *existing* site mapping tools. But that
> doesn't mean that we can't extract structural or semantic insights using an
> algorithmic approach. Entity extraction, auto-taxonomy and various other
> classification tools we already have do pretty amazing things. The fact that
> they are not used for site mapping as they cost $250,000 and up is
> orthogonal to the argument.

Jan only asked for help with site mapping...  Not all projects call
for those other "amazing things."

> Let software do the mapping, let humans to do the interpreting and
> designing.

The point of this discussion is that existing software doesn't do site
mapping well. The few tools that exist all seem to be very limited and
flawed.

--
Lyle

--------------------------
Lyle Kantrovich
Blog: Croc O' Lyle
http://crocolyle.blogspot.com