Nieminen, P; Rucker, G; Miettunen, J; Carpenter, J; Schumacher, M Statistically significant papers in psychiatry were cited more often than others JOURNAL OF CLINICAL EPIDEMIOLOGY, 60 (9): 939-946 SEP 2007

[Eugene Garfield]

E-mail Address: pentti.nieminen at oulu.fi 

Author(s): Nieminen, P (Nieminen, Pentti); Rucker, G (Rucker, Gerta); 
Miettunen, J (Miettunen, Jouko); Carpenter, J (Carpenter, James); 
Schumacher, M (Schumacher, Martin) 

Title: Statistically significant papers in psychiatry were cited more 
often than others 

Source: JOURNAL OF CLINICAL EPIDEMIOLOGY, 60 (9): 939-946 SEP 2007 

Language: English 

Document Type: Article 

Author Keywords: citation bias; publication bias; review literature; 
psychiatry; bibliometrics; data interpretation 

Keywords Plus: PUBLICATION BIAS; CITATION BIAS; EMPIRICAL-EVIDENCE; 
RANDOMIZED-TRIALS; CLINICAL-RESEARCH; MEDICAL JOURNALS; IMPACT FACTORS; 
QUALITY; PERFORMANCE; INDICATORS 

Abstract: Objective: Citations by other researchers are important in the 
dissemination of research findings. We aimed to investigate whether 
preferential citation of statistically significant articles exists in the 
psychiatric literature.

Study Design and Settings: We analyzed all original research papers 
published in 1996 in four psychiatric journals. Using a standardized 
questionnaire, from each paper, we extracted the primary outcome and its 
statistical significance. The number of citations, excluding 
authors' "self-citations," received by April 2005 was obtained. Regression 
analysis was used to relate citation frequency to statistical 
significance, adjusting for confounders.

Results: Of 448 extracted papers, 368 used statistical significance 
testing and 287 (77.8%) reported P < 0.05. The median number of citations 
for papers reporting "significant" and "nonsignificant" results was 33 vs. 
16. After adjustment for journal, study design, reporting quality, whether 
outcome confirmed previous findings and study size, the ratio of the 
number of citations per article for articles reporting "P < 0.05" on the 
primary outcome to those reporting "P > 0.05" was 1.63 (95% confidence 
interval [CI]: 1.32, 2.02, P < 0.001).

Conclusion: Authors cite studies based on their P-value rather than 
intrinsic scientific merit. This practice skews the research evidence. 
Systematic study registration and inclusion in meta-analysis should be 
encouraged. (c) 2007 Elsevier Inc. All rights reserved. 

Addresses: Oulu Univ, Med Informat Grp, FIN-90014 Oulu, Finland; Univ 
Freiburg, Inst Med Biometry & Med Informat, D-79115 Freiburg, Germany; 
Oulu Univ, Dept Psychiat, FIN-90014 Oulu, Finland 

Reprint Address: Nieminen, P, Oulu Univ, Med Informat Grp, POB 
5000,Aapistie 5A, FIN-90014 Oulu, Finland.
 
E-mail Address: pentti.nieminen at oulu.fi 

Cited Reference Count: 37 

Times Cited: 0 

Publisher: PERGAMON-ELSEVIER SCIENCE LTD 

Publisher Address: THE BOULEVARD, LANGFORD LANE, KIDLINGTON, OXFORD OX5 
1GB, ENGLAND 

ISSN: 0895-4356 

29-char Source Abbrev.: J CLIN EPIDEMIOL 

ISO Source Abbrev.: J. Clin. Epidemiol. 

Source Item Page Count: 8 

Subject Category: Public, Environmental & Occupational Health 

ISI Document Delivery No.: 202VI 

ALTMAN DG
Poor-quality medical research - What can journals do? 
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 287 : 2765 2002 

ALTMAN DG
STATISTICS IN MEDICAL JOURNALS - DEVELOPMENTS IN THE 1980S 
STATISTICS IN MEDICINE 10 : 1897 1991 

BEGG CB
PUBLICATION BIAS - A PROBLEM IN INTERPRETING MEDICAL DATA 
JOURNAL OF THE ROYAL STATISTICAL SOCIETY SERIES A-STATISTICS IN SOCIETY 
151 : 419 1988 

CALLAHAM M
Journal prestige, publication bias, and other characteristics associated 
with citation of published studies in peer-reviewed journals 
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 287 : 2847 2002 

CARTER AO
A survey identified publication bias in the secondary literature 
JOURNAL OF CLINICAL EPIDEMIOLOGY 59 : 241 DOI 
10.1016/j.jclinepi.2005.08.011 2006 

CHAN AW
Empirical evidence for selective reporting of outcomes in randomized 
trials - Comparison of Protocols to published articles 
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 291 : 2457 2004 

CHRISTENSENSZAL.JJ
THE CITATION BIAS - FAD AND FASHION IN THE JUDGMENT AND DECISION 
LITERATURE 
AMERICAN PSYCHOLOGIST 39 : 75 1984 

DICKERSIN K
THE EXISTENCE OF PUBLICATION BIAS AND RISK-FACTORS FOR ITS OCCURRENCE 
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 263 : 1385 1990 

EASTERBROOK PJ
PUBLICATION BIAS IN CLINICAL RESEARCH 
LANCET 337 : 867 1991 

EGGHE L
INTRO INFORMETRICS Q : 1990 

GARFIELD E
CITATION INDEXING TH : 1983 

GLANZEL W
A bibliometric approach to the role of author self-citations in scientific 
communication 
SCIENTOMETRICS 59 : 63 2004 

GLUUD LL
Bias in clinical intervention research 
AMERICAN JOURNAL OF EPIDEMIOLOGY 163 : 493 DOI 10.1093/aje/kwj069 2006 

GOTZSCHE PC
REFERENCE BIAS IN REPORTS OF DRUG TRIALS 
BRITISH MEDICAL JOURNAL 295 : 654 1987 

HOWARD L
Impact factors of psychiatric journals 
BRITISH JOURNAL OF PSYCHIATRY 170 : 109 1997 

HUTCHINSON BG
COMPREHENSIVENESS AND BIAS IN REPORTING CLINICAL-TRIALS - STUDY OF REVIEWS 
OF PNEUMOCOCCAL VACCINE EFFECTIVENESS 
CANADIAN FAMILY PHYSICIAN 41 : 1356 1995 

IOANNIDIS JPA
Molecular bias 
EUROPEAN JOURNAL OF EPIDEMIOLOGY 20 : 739 DOI 10.1007/s10654-005-2028-1 
2005 

IOANNIDIS JPA
Early extreme contradictory estimates may appear in published research: 
The Proteus phenomenon in molecular genetics research and randomized 
trials 
JOURNAL OF CLINICAL EPIDEMIOLOGY 58 : 543 DOI 
10.1016/j.jclinepi.2004.10.019 2005 

IOANNIDIS JPA
Contradicted and initially stronger effects in highly cited clinical 
research 
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 294 : 218 2005 

IOANNIDIS JPA
Genetic associations: false or true? 
TRENDS IN MOLECULAR MEDICINE 9 : 135 DOI 10.1016/S1471-4914(03)00030-3 
2003 

KJAERGARD LL
Citation bias of hepato-biliary randomized clinical trials 
JOURNAL OF CLINICAL EPIDEMIOLOGY 55 : 407 2002 

KOREN G
BIAS AGAINST NEGATIVE STUDIES IN NEWSPAPER REPORTS OF MEDICAL-RESEARCH 
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 266 : 1824 1991 

MCCULLAGH P
GEN LINEAR MODELS : 1989 

MIETTUNEN J
Statistical methodology in general psychiatric journals 
NORDIC JOURNAL OF PSYCHIATRY 56 : 223 2002 

MIETTUNEN J
The effect of statistical methods and study reporting characteristics on 
the number of citations: A study of four general psychiatric journals 
SCIENTOMETRICS 57 : 377 2003 

MOED HF
CITATION ANAL RES EV : 2005 

NIEMINEN P
Type of empirical research reports, as an explanatory factor in citation 
performance of psychiatric research 
SCIENTOMETRICS 35 : 309 1996
 
OJASOO T
Citation bias in medical journals 
SCIENTOMETRICS 45 : 81 1999 

PANTEL J
The evaluation of research performance in psychiatry. Potential and 
pitfalls of bibliometric analysis 
NERVENARZT 70 : 281 1999 

PERSSON O
Inflationary bibliometric values: The role of scientific collaboration and 
the need for relative indicators in evaluative studies 
SCIENTOMETRICS 60 : 421 2004 

RAVNSKOV U
CHOLESTEROL LOWERING TRIALS IN CORONARY HEART-DISEASE - FREQUENCY OF 
CITATION AND OUTCOME 
BRITISH MEDICAL JOURNAL 305 : 15 1992 

SCHULZ KF
EMPIRICAL-EVIDENCE OF BIAS - DIMENSIONS OF METHODOLOGICAL QUALITY 
ASSOCIATED WITH ESTIMATES OF TREATMENT EFFECTS IN CONTROLLED TRIALS 
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 273 : 408 1995 

SEGLEN PO
Citations and journal impact factors: questionable indicators of research 
quality 
ALLERGY 52 : 1050 1997 

SHADISH WR
AUTHOR JUDGMENTS ABOUT WORKS THEY CITE - 3 STUDIES FROM PSYCHOLOGY 
JOURNALS 
SOCIAL STUDIES OF SCIENCE 25 : 477 1995 

SNYDER H
Patterns of self-citation across disciplines (1980-1989) 
JOURNAL OF INFORMATION SCIENCE 24 : 431 1998 

SONG F
HEALTH TECHNOL ASSES 4 : 10 2000 

STERLING TD
PUBLICATION DECISIONS REVISITED - THE EFFECT OF THE OUTCOME OF STATISTICAL 
TESTS ON THE DECISION TO PUBLISH AND VICE-VERSA 
AMERICAN STATISTICIAN 49 : 108 1995