Does the specification of uncertainty hurt the progress of science?

[Jesper Wiborg Schneider]

Dear Loet,

I certainly do not argue against the "specification of uncertainty" - on the contrary I endorse "estimation" and "confidence intervals" - what I do point out is that "specification of uncertainty" indeed have meta-theoretical underpinnings and that it comes with assumptions. If the assumptions are not fulfilled, such as randomness, "specification of uncertainty" becomes a rather meaningless task.

I give some suggestions in the paper (and below) for best practice if one has a frequentist approach to inferential statistics - it should be clear from these that I have no quarrels with the "specification of uncertainty" - what worries me is the way inferential statistics are most often practiced and the epistemic weight we give the results even when the assumptions are clearly violated.

A preprint of the accepted version of the paper will be available tomorrow at the arXiv

1) statistical inference only makes sense when data come from a probability sample or have been randomly assigned to treatment and control groups;
2) whenever possible take an estimation framework, starting with the formulation of research aims such as "how much?" or "to what extent?";
3) interpretation of research results should be based on point and interval estimates;
4) calculate effect size estimates and confidence intervals to answer those questions, then interpret results based on informed judgment;
5) if statistical significance tests are used, (a) information on power must be reported, and (b) the null hypothesis should be plausible;
6) effect sizes and confidence intervals must be reported whenever possible for all effects studied, whether large or small, statistically significant or not;
7) exact p values should be reported;
8) it is unacceptable to describe results solely in terms of statistical significance;
9) use the word "significant" without the qualifier "statistically" only to describe results that are truly noteworthy;
10) it is the researcher's responsibility to explain why the results have substantive significance; statistical tests are inadequate for this purpose;
11) replication is the best way to deal with sampling error.

Kind regards - Jesper

_____________________

Jesper W. Schneider
Senior Researcher, PhD

Aarhus University
Business and Social Sciences
Danish Centre for Studies in Research & Research Policy,
Department of Political Science & Government

Finlandsgade 4
DK-8200 Aarhus N
Denmark

T: +45 8716 5241
M: jws at cfa.au.dk<mailto:jws at cfa.au.dk>
W: http://pure.au.dk/portal/en/jws@cfa.au.dk

[bslogo_mail_equis_stor_uk]



From: loet at leydesdorff.net [mailto:leydesdorff at gmail.com] On Behalf Of Loet Leydesdorff
Sent: 25 September 2012 08:41
To: SIGMETRICS at LISTSERV.UTK.EDU
Cc: Jesper Wiborg Schneider
Subject: Does the specification of uncertainty hurt the progress of science?

Does the specification of uncertainty hurt the progress of science?<http://arxiv.org/abs/1209.5272>
In "Caveats for using statistical significance tests in research assessments,"--forthcoming in the Journal of Informetrics, but available at arXiv:1112.2516<http://arxiv.org/abs/1112.2516> -- Schneider (2012) focuses on Opthof & Leydesdorff (2010) as an example of the misuse of statistics in the social sciences. However, our conclusions are theoretical since they are not dependent on the use of one statistics or another. We agree with Schneider insofar as he proposes to develop further statistical instruments (such as effect sizes). Schneider (2012), however, argues on meta-theoretical grounds against the specification of uncertainty because, in his opinion, the presence of statistics would legitimate decision-making. We disagree: uncertainty can also be used for opening a debate. Scientometric results in which error bars are suppressed for meta-theoretical reasons should not be trusted.

Loet Leydesdorff & Tobias Opthof
(Submitted on 24 Sep 2012; available at http://arxiv.org/abs/1209.5272.)

** apologies for cross-postings
________________________________
Loet Leydesdorff
Professor, University of Amsterdam
Amsterdam School of Communications Research (ASCoR)
Kloveniersburgwal 48, 1012 CX Amsterdam.
Tel. +31-20-525 6598; fax: +31-842239111
loet at leydesdorff.net <mailto:loet at leydesdorff.net> ; http://www.leydesdorff.net/
Visiting Professor, ISTIC, <http://www.istic.ac.cn/Eng/brief_en.html> Beijing; Honorary Fellow, SPRU, <http://www.sussex.ac.uk/spru/> University of Sussex; http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20120925/9e94f501/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 6171 bytes
Desc: image001.jpg
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20120925/9e94f501/attachment.jpg>