[Sigia-l] Making assumptions

[Jonathan Baker-Bates]

On Sun, 2009-08-02 at 04:13 -0400, Ziya Oz wrote: 
> > the results can be interpreted in a controlled manner
> 
> This is the most egregious part of 'design research'.
> 95% of the time there is no 'control'.
> Except for an identical A/B test, it's as scientific as a Ouija Board.

I didn't mean to use the word "controlled" in the sense of comparing the
results of a test. I meant that I could prevent the results from being
interpreted by others in ways I did not want. Apologies for the
confusion there.

> A lot of charlatans live off the illusion of such numerical 'confirmation.'
> And when you can do an A/B test of sufficient statistical significance, the
> result is often a feedback of extremely narrow, tactical detail, as it
> should be.

I don't think anything I've said would contradict that. I'm only
proposing an indication of "strength" around either specific or general
design assumptions. Assumptions that will be challenged by those pesky
stakeholders ;-)

> Design is the correlation *among* such details at a strategic level.
> Try measuring that. :)

So am I to understand that with the exception of multivariate tests, you
would not consider design research at all? 

> Proof is, obviously, in the pudding of achieving the intent/business goal.

I'm not sure it is obvious, is it? Whatever the business goal, you can
always hit it harder, and there will never be any shortage of people in
the business who think they have a way of doing that - hence my concern
for control.

Jonathan