[Sigia-l] Findability

[Karl Fast]

> > My opinion is that (a) a balance is required and (b) that balance will
> > keep shifting in favor of algorithms because algorithms will evolve at
> > a much faster pace than we will.

> I agree with your overall perspective here, but could you clarify your
> theory on the evolution of algorithms? 


Simon pretty much covered it, but let me respond from a slightly
different angle.

Consider the evolution of information retrieval over the last few
decades where we have gone from the foundations laid down by Salton
to developments like Google and Teoma.

Imagine it's 1983, Reagan is still in his first term, and I said
that in twenty years it would be commonplace to search through a
three billion document set in less a second and return results that
most people were satisfied with most of the time. Not perfect mind
you, but good enough that the company offering this service would
find their name being turned into a verb and being suggested as a
new word in the OED.

If I said this you'd think I was bonkers. But you'd be wrong.

Google and Teoma represent significant evolutionary advances in
algorithm development. And that's just information retrieval.

In the same time frame (two decades), how far have human beings
advanced in our abilities to classify and organize information? 

There are astounding developments happening in areas like genetic
algorithms. Consider this recent story in Discover:

   http://www.discover.com/aug_03/gthere.html?article=feattech.html

And the related video of a software biped learning how to walk
based on an evolutionary genetic algorithm:

   http://www.naturalmotion.com/pages/technology_hiw.htm


I am not saying that humans do not evolve. Nor am I saying that
algorithms will replace people, there is no need for humans in terms
of organizing information, or that we better start preparing for a
Terminator/Matrix scenario.

To say that would be like sitting in 19th century England, watching
the rapid development of the steam engine and other mechanical
devices, and concluding that human beings were doomed because we
weren't evolving as fast as the machines were (however, if you've
read "The Muse in the Machine," by David Gelernter, a book about
artificial intelligence, you may remember that he argues that humans
have evolved significantly in the last few thousand years, and I
think his point is excellent).

I'm not saying all that doom-n-gloom stuff.

But what I am saying is that algorithms are developing quickly and
that this rapid development will continue to shift the balance
between what humans do and machines do. This shift is accelerating
because algorithms are only getting better. And they are getting
better really fast.

Meanwhile, humans are pretty much the same as we've always been. If
we are evolving (and I think we are), it's happening at a much
slower pace and in different directions than, for example, algorithms.

As Simon notes, humans don't scale well. So we are abdicating the
job of sifting through huge piles of information to algorithms. This
makes sense.

But computers don't understand well. Humans do. Google doesn't
understand what it finds. Humans do.

It's been noted that the purpose of computing is insight, not
numbers. But it's important to understand that humans are the ones
who develop the insight. The computer still just sees numbers, no
matter the algorithm. Humans develop insight. Computers just run the
algorithm.
 

Of course, if someone makes a cyber-librarian like in Neal
Stephenson's "Snow Crash" then it's game over.


--karl