[Sigia-l] Re: Eye-tracking systems: is it worth the price?

[Gael Laurans]

Hi Eryk,

You're asking the right question indeed : eye tracking is a very exciting technology but one should carefully evaluate if it's really worth the price. The cost of eye-tracking lies not only in the cost of the equipment itself but also in the effort needed to use it.

There are a lot of time-consuming technical problems you have to deal with and a potentially huge quantity of data being produced. With the same time and budget it means you will have less ressources available to carefully think, prepare and analyse your tests.

You also have to expect eye tracking failures with some test participants (make-up, iris color, some type of glasses all might be a problem depending on your equipment, light in the room is important), which also means you're going to need a bigger sample to get meaningful data.

During the test you also need to calibrate the device and to ask your subject to be realtively static. Assuming you're testing software, modern systems with a standard accuracy (50 Hz, 1 degree of spatial resolution) do not require the subject to be perfectly immobile or to wear an equipment but they nonetheless need that the subject's head stays in a limited space. If the test lasts one or two hours you might have to ask your subject to sit up again and again.

Then because scrolling and differing paths between users might cause problems for the eye tracking software, you are sometimes forced to ask your test participants to passively look at your homepage before giving them any task. Data collected in this way might not be very relevant to understand the attentional behaviour of a real, purpose-oriented users.

Now, what can you expect from eye-tracking ? Basically it allows you to distinguish between navigation elements that have been gazed at (and presumably seen and attended to) and those which have not (and presumably not been perceived at all). So if nobody is clicking the link that would help them find the information they're looking for, eye tracking might tell you why (not seeing it at all ? not looking in the right place ? not understanding the relevance ?). 

However it is often the case that you are able to guess that by looking at the user's behaviour, by probing for it (even if every psychologist knows the limitation of self-report data) or in reference to a well-documented phenomenon like banner blindness. In any case, all uses of eye-tracking in usability I have seen to date are concerned with very low-level interaction.

Regarding literature, Andrew Duchowski's book Eye Tracking Tool and Methodology is highly recommended, comprehensive yet very readable (you can skip the more technical chapters if you like).

If you had to read just one thing, I would recommend this note by the belgian company Namahn : http://www.namahn.com/resources/documents/note-eyetracking.pdf

An interesting article from David Richardson and Michael Spivey on the website of the former http://psychology.stanford.edu/~richardson/docs/EyeTrackingEBBE.pdf

Eye Tracking in Human-Computer Interaction and Usability Research: Ready to Deliver the Promises from Robert Jacobs and Keith Karn is available at http://www.cs.tufts.edu/~jacob/papers/ecem.pdf

The famous Poynter studies : http://www.poynterextra.org/et/i.htm and a comment from Jakob Nielsen : http://www.useit.com/alertbox/20000514.html

An article from UIE on the subject : http://www.uie.com/articles/eye_tracking/

A case study from the german company eye square :
http://www.eye-square.com/documents/EyeTracking-ResearchApplications.pdf

Another critical view on eye tracking for usability : Eye Tracking in Usability: Is It Worthwile ? by Antti Aaltonen (http://www.cs.uta.fi/hci/mulmod/material/etusab.pdf)

The bottom line : eye tracking is interesting for very well-funded projects or as a scientific tool but probably not suited to be routinely used in usability tests. Now if you have clients ready to pay for the extra thrill in the power point presentation...

Gael Laurans