[Sigia-l] What's your score?

[Karl Fast]

> > But I labelled all messages phishing frauds if they came from an
> > organization I didn't do business with. I didn't even look at the
> > message in these cases.
> 
> That's not a very reliable assumption for the real world. People do
> indeed respond to email from all sorts of originations. 

True. People do respond to all kinds of email, which is why phishing
works. But the test is based on the assumption that people always
open messages to determine if it's legitimate or not. Not true.

The instructions were "If you received one of these emails in your
inbox, what would you do?"

Using the sender and subject information, I divide new messages into
three rough categories (I think most people do something similar):

1. Messages I know are legitimate. At least, I believe there is a
   high probability they are legitimate. I open and read these
   messagess.

2. Messages I know are spam or phishing. At least, I believe there
   is high probability they are not legitimate. I delete these
   messages without opening them, if the probability is high enough.

3. Messages I am not sure about. They may be legitimate. They may
   not. I open these messages to be sure. 

I suspect most people do a similar triage, though each person will
do this triage based on different criteria. For example, one of my
criteria relates to the sender. Is the message is from an
organization I recognize? Is the organization one I do business
with? Does the organization communicate with me by email or not? For
example, I know that PayPal and Ebay don't send me email, except
when it pertains to certain transactions that I have initiated.

Of course, everyone uses different criteria and there are probably
people who open every single message. Maybe they like getting
investment opportunities from Nigeria. I do not.

The test covers only situations 1 and 3. That is, the situation when
people actually open the message.

This is not the problem with the test.

The problem is that

 (a) people regularly make decisions based on the sender and/or
     subject information (who knows how many or how often?)

 (b) and the test doesn't deal with that case

 (c) so the results are easy to misinterpret

That last point is the key one.
 
The press release says "The average score of 75 percent indicates
that one fourth of email users today still are unable to
consistently distinguish between legitimate and phishing email."
http://www.mailfrontier.com/press/press_fieldguide.jsp

It should say "the average score of 75 percent indicates that,
***for messages they have opened***, one fourth of email users today
are unable to consistently distinguish between legitimate and
phishing email." 

If we accept the hypothesis that people make decisions without
opening the message, then the results are almost certainly skewed. I
don't know if they are skewed high or low. Or how much they are
skewed. But they will be skewed.

A more comprehensive test would also deal with the "I'm not going to
open that message" behavior.

Still, the test is probably quite effective at raising awareness of
how difficult it can be to identify phishing attempts. And making it
more comprehensive would make it more complex and reduce the number
of respondents.

Regardless, I would be careful about how I quote that 75% figure.


-- 
Karl Fast
http://www.livingskies.com/