Weiss, AP (Weiss, Anthony P.) Measuring the impact of medical research: Moving from outputs to outcomes AMERICAN JOURNAL OF PSYCHIATRY, 164 (2): 206-214 FEB 2007

[Eugene Garfield]

E-mail Address: aweiss at partners.org 

Author(s): Weiss, AP (Weiss, Anthony P.) 

Title: Measuring the impact of medical research: Moving from outputs to 
outcomes 

Source: AMERICAN JOURNAL OF PSYCHIATRY, 164 (2): 206-214 FEB 2007 

Language: English 

Document Type: Review 

Keywords Plus: JOURNAL READING HABITS; HOPKINS-UNIVERSITY SCHOOL; CLINICAL-
TRIALS; PUBLIC TRUST; HEALTH-CARE; QUESTIONNAIRE SURVEY; BIOMEDICAL-
RESEARCH; ACADEMIC PROMOTION; ANXIETY DISORDERS; MENTAL-ILLNESS 

Abstract: Billions of dollars are spent every year to support medical 
research, with a substantial percentage coming from charitable 
foundations. To justify these expenditures, some measure of the return on 
investment would be useful, particularly one aligned with the intended 
ultimate outcome of this scientific effort: the amelioration of disease. 
The current mode of reporting on the success of medical research is output 
based, with an emphasis on measurable productivity. This approach falls 
short in many respects and may be contributing to the well-described 
efficacy-effectiveness gap in clinical care. The author argues for an 
outcomes-based approach and describes the steps involved, using an 
adaptation of the logic model. A shift in focus to the outcomes of our 
work would provide our funders with clearer mission-central return-on-
investment feedback, would make explicit the benefits of science to an 
increasingly skeptical public, and would serve as a compass to guide the 
scientific community in playing a more prominent role in reducing the 
efficacy-effectiveness gap. While acknowledging the enormous complexity 
involved with the implementation of this approach on a large scale, the 
author hopes that this essay will encourage some initial steps toward this 
aim and stimulate further discussion of this concept. 

Addresses: Massachusetts Gen Hosp, Dept Psychiat, Boston, MA 02114 USA; 
Harvard Univ, Sch Med, Cambridge, MA 02138 USA 

Reprint Address: Weiss, AP, Dept Psychiat, MGH E,Bldg 149,Rm 2606,13th St, 
Charlestown, MA 02129 USA. 

E-mail Address: aweiss at partners.org 

Cited Reference Count: 100 

Times Cited: 0 

Publisher: AMER PSYCHIATRIC PUBLISHING, INC 

Publisher Address: 1000 WILSON BOULEVARD, STE 1825, ARLINGTON, VA 22209-
3901 USA 

ISSN: 0002-953X 

29-char Source Abbrev.: AMER J PSYCHIAT 

ISO Source Abbrev.: Am. J. Psychiat. 

Source Item Page Count: 9 

Subject Category: Psychiatry 

ISI Document Delivery No.: 131XM 

*I MED
CROSS QUAL CHASM NEW : 2001 

*MILB MEM FUND
BETT INF BETT OUTC U : 2000 

*UN WAY
MEAS PROGR OUTC PRAC : 1996 

ARNDT KA
INFORMATION EXCESS IN MEDICINE - OVERVIEW, RELEVANCE TO DERMATOLOGY, AND 
STRATEGIES FOR COPING 
ARCHIVES OF DERMATOLOGY 128 : 1249 1992 

ASAI A
Focus group interviews examining attitudes towards medical research among 
the Japanese: A qualitative study 
BIOETHICS 18 : 448 2004 

ATASOYLU AA
Promotion criteria for clinician-educators 
JOURNAL OF GENERAL INTERNAL MEDICINE 18 : 711 2003 

AVORN J
SCIENTIFIC VERSUS COMMERCIAL SOURCES OF INFLUENCE ON THE PRESCRIBING 
BEHAVIOR OF PHYSICIANS 
AMERICAN JOURNAL OF MEDICINE 73 : 4 1982 

BATSHAW ML
ACADEMIC PROMOTION AT A MEDICAL-SCHOOL - EXPERIENCE AT JOHNS-HOPKINS-
UNIVERSITY SCHOOL OF MEDICINE 
NEW ENGLAND JOURNAL OF MEDICINE 318 : 741 1988 

BAUER MS
Principles of effectiveness trials and their implementation in VA 
cooperative study #430: 'Reducing the efficacy-effectiveness gap in 
bipolar disorder' 
JOURNAL OF AFFECTIVE DISORDERS 67 : 61 2001 

BERTOLINO A
Interaction of COMT Val(108/158) met genotype and olanzapine treatment on 
prefrontal cortical function in patients with schizophrenia 
AMERICAN JOURNAL OF PSYCHIATRY 161 : 1798 2004 

BERWICK DM
Disseminating innovations in health care 
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 289 : 1969 2003 

BOULWARE LE
Race and trust in the health care system 
PUBLIC HEALTH REPORTS 118 : 358 2003 

BROUSSEAU RT
Essay - Are foundations overlooking mental health? 
HEALTH AFFAIRS 22 : 222 DOI 10.1377/hlthaff.22.5.222 2003 

BRUNT ME
Mass media release of medical research results - An analysis of 
antihypertensive drug prescribing in the aftermath of the calcium channel 
blocker scare of March 1995 
JOURNAL OF GENERAL INTERNAL MEDICINE 18 : 84 2003 

BURKE DT
Reading habits of physical medicine and rehabilitation resident physicians 
AMERICAN JOURNAL OF PHYSICAL MEDICINE & REHABILITATION 83 : 551 DOI 
10.1097/01.PHM.0000130035.54932.B9 2004 

BURKE DT
Reading habits of practicing physiatrists 
AMERICAN JOURNAL OF PHYSICAL MEDICINE & REHABILITATION 81 : 779 DOI 
10.1097/01.PHM.0000027420.36815.3F 2002 

BUXTON M
Estimating the economic value to societies of the impact of health 
research: a critical review 
BULLETIN OF THE WORLD HEALTH ORGANIZATION 82 : 733 2004 

BUXTON M
J HLTH SERV RES POLI 1 : 35 1996 

BUXTON M
Assessing benefits from Department of Health and National Health Service 
Research & Development 
PUBLIC MONEY & MANAGEMENT 20 : 29 2000 

CALVO BC
J AM BOARD FAM PRACT 15 : 457 2002 

CAPLOW T
HARVARD BUS REV 30 : 105 1954 

CASALINO L
External incentives, information technology, and organized processes to 
improve health care quality for patients with chronic diseases 
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 289 : 434 2003 

CHARNEY DS
Increasing public trust and confidence in psychiatric research 
BIOLOGICAL PSYCHIATRY 46 : 1 1999 

CLARKE GN
IMPROVING THE TRANSITION FROM BASIC EFFICACY RESEARCH TO EFFECTIVENESS 
STUDIES - METHODOLOGICAL ISSUES AND PROCEDURES 
JOURNAL OF CONSULTING AND CLINICAL PSYCHOLOGY 63 : 718 1995 

COMROE JH
SCIENTIFIC BASIS FOR SUPPORT OF BIOMEDICAL SCIENCE 
SCIENCE 192 : 105 1976 

COOMARASAMY A
HLTH INFORMATION LIB 18 : 183 2001 

CORBIESMITH G
Distrust, race, and research 
ARCHIVES OF INTERNAL MEDICINE 162 : 2458 2002 

CURTIS NM
Multisystemic treatment: A meta-analysis of outcome studies 
JOURNAL OF FAMILY PSYCHOLOGY 18 : 411 DOI 10.1037/0893-3200.18.3.411 2004 

 FASSOULAKI A
Academic anesthesiologists' views on the importance of the impact factor 
of scientific journals: a North American and European survey 
CANADIAN JOURNAL OF ANAESTHESIA-JOURNAL CANADIEN D ANESTHESIE 48 : 953 
2001 

FERNANDEZCANO A
Reconsidering Price's model of scientific growth: An overview 
SCIENTOMETRICS 61 : 301 2004 

FINEBERG HV
ASSESSING MED TECHNO : 1985 

FINEBERG HV
CLINICAL-EVALUATION - HOW DOES IT INFLUENCE MEDICAL-PRACTICE 
BULLETIN DU CANCER 74 : 333 1987 

GAMBLE VN
A LEGACY OF DISTRUST - AFRICAN-AMERICANS AND MEDICAL-RESEARCH 
AMERICAN JOURNAL OF PREVENTIVE MEDICINE 9 : 35 1993 

GARFIELD E
CURR CONTENTS 25 : 3 1994 

GARFIELD E
THE ASIS OUTSTANDING INFORMATION-SCIENCE TEACHER AWARD - 
LANCASTER,F.WILFRID IS THE 1ST WINNER 
CURRENT CONTENTS : 5 1981 

GARFIELD E
EXPANSION OF CITATION CLASSICS - 250 UNIQUE COMMENTARIES PER YEAR 
CURRENT CONTENTS 45 : 5 1979 

GJERDE C
FAM MED 26 : 361 1994 

GLASGOW RE
Why don't we see more translation of health promotion research to 
practice? Rethinking the efficacy-to- effectiveness transition 
AMERICAN JOURNAL OF PUBLIC HEALTH 93 : 1261 2003 

GRANT J
Evaluating "payback" on biomedical research from papers cited in clinical 
guidelines: applied bibliometric study 
BRITISH MEDICAL JOURNAL 320 : 1107 2000 

GRANT RW
Clinical inertia in the management of Type 2 diabetes metabolic risk 
factors 
DIABETIC MEDICINE 21 : 150 2004 

GRUMAN J
Essay - Health research philanthropy in a time of plenty: A strategic 
agenda 
HEALTH AFFAIRS 21 : 265 2002 

HAMILTON DP
PUBLISHING BY - AND FOR QUESTIONABLE - THE NUMBERS 
SCIENCE 250 : 1331 1990 

HANNEY S
HLTH RES POLICY SYST 2 : 4 2004 

HOLDFORD DA
J AM PHARM ASSOC 44 : 387 2004 

JONES T
What British psychiatrists read - Questionnaire survey of journal usage 
among clinicians 
BRITISH JOURNAL OF PSYCHIATRY 185 : 251 2004 

JORM AF
Beliefs about the helpfulness of interventions for mental disorders: a 
comparison of general practitioners, psychiatrists and clinical 
psychologists 
AUSTRALIAN AND NEW ZEALAND JOURNAL OF PSYCHIATRY 31 : 844 1997 

KATON WJ
Quality of depression care in a population-based sample of patients with 
diabetes and major depression 
MEDICAL CARE 42 : 1222 2004 

KELCH RP
Maintaining the public trust in clinical research. 
NEW ENGLAND JOURNAL OF MEDICINE 346 : 285 2002 

KENNEDY D
Clinical trials and public trust 
SCIENCE 306 : 1649 DOI 10.1126/science.1107657 2004 

LAURITSEN J
Publications in anesthesia journals: Quality and clinical relevance 
ANESTHESIA AND ANALGESIA 99 : 1486 DOI 10.1213/01.ANE.0000134468.92993.CF 
2004 

LAVIS J
J HLTH SERV RES POLI 8 : 165 2003 

LEE KP
Association of journal quality indicators with methodological quality of 
clinical research articles 
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 287 : 2805 2002 

LEHMAN AF
Patterns of usual care for Schizophrenia: Initial results from the 
Schizophrenia Patient Outcomes Research Team (PORT) client survey 
SCHIZOPHRENIA BULLETIN 24 : 11 1998 

LEVINSON W
Mission critical - Integrating clinician-educators into academic medical 
centers 
NEW ENGLAND JOURNAL OF MEDICINE 341 : 840 1999 

LEWISON G
>From biomedical research to health improvement 
SCIENTOMETRICS 54 : 179 2002 

LEWISON G
Researchers' and users' perceptions of the relative standing of biomedical 
papers in different journals 
SCIENTOMETRICS 53 : 229 2002 

MCHUGO GJ
Fidelity to assertive community treatment and client outcomes in the New 
Hampshire dual disorders study 
PSYCHIATRIC SERVICES 50 : 818 1999 

MCNEIL BJ
Shattuck lecture - Hidden barriers to improvement in the quality of care 
NEW ENGLAND JOURNAL OF MEDICINE 345 : 1612 2001 

MINTZES B
Influence of direct to consumer pharmaceutical advertising and patients' 
requests on prescribing decisions: two site cross sectional survey 
BRITISH MEDICAL JOURNAL 324 : 278 2002 

MOLASGALLART J
Assessing the non-academic impact of grant-funded socio-economic research: 
results from a pilot study 
RESEARCH EVALUATION 9 : 171 2000 

MURPHY K
Diminishing returns? the costs and benefits of improving health 
PERSPECTIVES IN BIOLOGY AND MEDICINE 46 : S108 2003 

MURRAY CJL
Global mortality, disability, and the contribution of risk factors: Global 
Burden of Disease Study 
LANCET 349 : 1436 1997 

NAGPAUL PS
Constructing a multi-objective measure of research performance 
SCIENTOMETRICS 56 : 383 2003 

PASSELL P
EXCEPTIONAL RETURNS : 2000 

PATTERSON RE
Is there a consumer backlash against the diet and health message? 
JOURNAL OF THE AMERICAN DIETETIC ASSOCIATION 101 : 37 2001 

PENDLEBURY DA
SCIENCE, CITATION, AND FUNDING 
SCIENCE 251 : 1410 1991 

PINCUS HA
THE ANATOMY OF RESEARCH FUNDING OF MENTAL-ILLNESS AND ADDICTIVE DISORDERS 
ARCHIVES OF GENERAL PSYCHIATRY 49 : 573 1992 

PRICE DJ
LITTLE SCI BIG SCI : 1986 

RAUCH SL
COMPREHENSIVE TXB PS : 2983 2005 

ROBINSON JC
The end of asymmetric information 
JOURNAL OF HEALTH POLITICS POLICY AND LAW 26 : 1045 2001 

ROSENBERG LE
Exceptional economic returns on investments in medical research 
MEDICAL JOURNAL OF AUSTRALIA 177 : 368 2002 

ROYBYRNE PP
A Randomized effectiveness trial of cognitive-behavioral therapy and 
medication for primary care panic disorder 
ARCHIVES OF GENERAL PSYCHIATRY 62 : 290 2005 

ROYBYRNE PP
Moving treatment research from clinical trials to the real world 
PSYCHIATRIC SERVICES 54 : 327 2003 

SAHA S
Impact factor: a valid measure of journal quality? 
JOURNAL OF THE MEDICAL LIBRARY ASSOCIATION 91 : 42 2003 

SAINT S
Journal reading habits of internists 
JOURNAL OF GENERAL INTERNAL MEDICINE 15 : 881 2000 

SCHEIN M
CURR SURG 57 : 252 2000 

SCHOENWALD SK
Effectiveness, transportability, and dissemination of interventions: What 
matters when? 
PSYCHIATRIC SERVICES 52 : 1190 2001 

SKINNER K
J CONTIN EDUC NURS 20 : 170 1989 

STEIN MB
Quality of care for primary care patients with anxiety disorders 
AMERICAN JOURNAL OF PSYCHIATRY 161 : 2230 2004 

STEYER TE
Medicaid reimbursement for alternative therapies 
ALTERNATIVE THERAPIES IN HEALTH AND MEDICINE 8 : 84 2002 

STROSS JK
DISSEMINATION OF NEW MEDICAL INFORMATION 
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 241 : 2622 1979 

SWARTZ MS
Assessing clinical and functional outcomes in the Clinical Antipsychotic 
Trials of Intervention Effectiveness (CATIE) schizophrenia trial 
SCHIZOPHRENIA BULLETIN 29 : 33 2003 

TALAMANCA AF
B GROUP INT RECH SCI 44 : 2 2002 

THOMAS PA
Results of an academic promotion and career path survey of faculty at the 
Johns Hopkins University School of Medicine 
ACADEMIC MEDICINE 79 : 258 2004 

TOMLINSON S
The research assessment exercise and medical research 
BRITISH MEDICAL JOURNAL 320 : 636 2000 

TORREY EF
PORT updated treatment recommendations 
SCHIZOPHRENIA BULLETIN 30 : 617 2004 

TRELLE S
Information management and reading habits of German diabetologists: a 
questionnaire survey 
DIABETOLOGIA 45 : 764 DOI 10.1007/s00125-002-0807-8 2002 

VICKERY CE
JOURNAL READING HABITS OF DIETITIANS 
JOURNAL OF THE AMERICAN DIETETIC ASSOCIATION 92 : 1510 1992 

VYDARENY KH
The road to success: Factors affecting the speed of promotion of academic 
radiologists 
ACADEMIC RADIOLOGY 6 : 564 1999 

WANG PS
Adequacy of treatment for serious mental illness in the United States 
AMERICAN JOURNAL OF PUBLIC HEALTH 92 : 92 2002 

WARE JE
THE MOS 36-ITEM SHORT-FORM HEALTH SURVEY (SF-36) .1. CONCEPTUAL-FRAMEWORK 
AND ITEM SELECTION 
MEDICAL CARE 30 : 473 1992 

WAZANA A
Physicians and the pharmaceutical industry - Is a gift ever just a gift? 
JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 283 : 373 2000 

WEISS B
THE IMPACT OF METHODOLOGICAL FACTORS ON CHILD-PSYCHOTHERAPY OUTCOME 
RESEARCH - A METAANALYSIS FOR RESEARCHERS 
JOURNAL OF ABNORMAL CHILD PSYCHOLOGY 18 : 639 1990 

WEISZ JR
BRIDGING THE GAP BETWEEN LABORATORY AND CLINIC IN CHILD AND ADOLESCENT 
PSYCHOTHERAPY 
JOURNAL OF CONSULTING AND CLINICAL PSYCHOLOGY 63 : 688 1995 

WELLS KB
Treatment research at the crossroads: The scientific interface of clinical 
trials and effectiveness research 
AMERICAN JOURNAL OF PSYCHIATRY 156 : 5 1999 

WEST JC
Patterns and quality of treatment for patients with schizophrenia in 
routine psychiatric practice 
PSYCHIATRIC SERVICES 56 : 283 2005 

WHOLEY J
EVALUATION PROMISE P : 1979 

WILSON P
UNDERTAKING SYSTEMAT : 2000 

WRIGHT SM
Motivating factors for academic physicians within departments of medicine 
MAYO CLINIC PROCEEDINGS 79 : 1145 2004 

YOUNG AS
The quality of care for depressive and anxiety disorders in the United 
States 
ARCHIVES OF GENERAL PSYCHIATRY 58 : 55 2001