[Sigia-l] Findability (long)

[Rashmi Sinha]

I have been following this discussion about Findability and have a few 
comments to add about Recommender Systems (that have been praised for 
helping people find things directly, without categories), and metadata 
search as compared to Google type search.

If you look closely at collaborative filtering algorithms (that form the 
basis of many recommender systems), then you will find that they are not as 
category-free as you think. The first step with many Recommender Systems 
algorithms is dimensionality reduction, which often means some sort of a 
clustering (you start getting close to categorization here, though of a 
different kind). Another way is to take genres into account (Note. I am 
more familiar with book, movie and music recommender systems). Recommending 
within genres (which is what Amazon mostly does) is a conservative 
technique designed to increase revenues as opposed to helping users 
discover more about their tastes (look at MediaUnbound for the best Music 
Recommnder system, ever!). Better Recommender Systems , which truly help 
users find new items, and expand their tastes, find much less of a market. 
I cannot repeat names of companies here, but I recently launched on a 
mission to find out what happened to many of the great Recommender Systems 
that were developed in the last few years. The story is the same, many 
Recommender System companies are finding it hard to sell their best 
products that really helped users explore their tastes. The Recommender 
Systems that sell, are ones that help companies make the sale, and make it 
fast. (We wrote about this contrast between systems that help users explore 
tastes and systems that help make the sale at 
http://www.rashmisinha.com/articles/musicDIS.pdf (Self link))

Also, it is incorrect to think that Recommender Systems cannot have an 
agenda, or less of an agenda than categorization. Recommender Systems are 
explicitly designed to encourage people to buy. Often, they are the 
technique that helps the telemarketer suggest another product to you in a 
late evening phone call. In contrast browse, or search systems are much 
more self-directed. Recommender System algorithms are fine tuned for 
marketing and sales purposes not for helping you discover information.

One quote that comes to mind about Recommender Systems: "When they are 
good, they are very very good, when they are bad, they are very very bad". 
Read this article in the Wall Street Journal Opinion Page to know what I 
mean: "If TiVo Thinks You Are Gay, Here's How to Set It Straight" 
http://online.wsj.com/article_email/0,,SB1038261936872356908,00.html

Don't think Recommender Systems do not use classification. Apart from other 
things, they also classify YOU. And they classify you without any knowledge 
or choice on your part. The privacy issues are huge. Although I am a big 
advocate of Recommender Systems, I understand that such systems can be used 
(and probably are being used) in a downright insidious manner. IMO, using a 
good Metadata browse and search seems like a walk in the park, where I can 
stop and smell the roses when I feel like it, as opposed to using a 
recommender system which, in the wrong hands, can be like being followed 
around by a detective. I like the openness of categorization techniques. It 
is possible for a user to see the scheme, the explore the biases of the 
scheme if interested. With a Recommender System, you have little access to 
what it thinks of you.

A different issue: Is good keyword search always better than metadata 
search? Read the latest paper from the Flamenco project which shows that at 
least for some cases, metadata search is preferred. 
http://bailando.sims.berkeley.edu/papers/flamenco-chi03.pdf

At the end: Why use categorization schemes to display information? For the 
very reason that human beings use categorical lenses to view the world (By 
the way, there is an inordinate amount of evidence from cognitive 
psychology, anthropology, neuroscience, neuropsychology that human beings 
indeed view the world through categorical lenses, and this ability emerges 
very early in life). It enables a more efficient filtering of information. 
It also provides a social glue by, and allows for collective 
representations, and distributed cognition. The key for the information 
architect is to find the correct categorical scheme to use. I would tend to 
agree with those who point out that a bad categorical scheme is worse than 
other options (because it actively misleads the user).

Research also shows that there is great deal of consensus in how people 
categorize within a culture (see research by Romney and Weller). 
Categorization allows us to tap into the domain knowledge of another person 
without needing to learn everything they know. Read this article on 
categories used by bartenders: http://www.lehigh.edu/~jbg1/persknow.htm

IMO, there is a great deal of promise in using the correct categorization 
schemes as and when it is appropriate.

TO end, here's an incident that illustrates what I have been trying to 
convey: One of the Recommender Systems I referred to above has helped me 
tremendously in expanding my musical tastes. It recommended a number of 
songs that I thoroughly enjoyed, but had not heard before. They all seemed 
to have something in common, but I was not sure what. It was only when it 
also identified the genre as Latin Jazz, that I knew how to tie this new 
found taste to other music genres I like, how to follow that interest out 
of the confines of the Recommender System.

Hope this is not too confusing!
-rashmi