Jacso P "Errors of omission and their implications for computing scientometric measures in evaluating the publishing productivity and impact of countries" Online Information Review 33(2):376-385, 2009

Mon Jul 20 16:40:52 EDT 2009

-----------------------------------------------------------
e-MAIL: jacso at hawaii.edu

TITLE :  Errors of omission and their implications for computing 
scientometric measures in evaluating the publishing productivity and impact 
of countries 

Author(s): Jacso P (Jacso, Peter)  

Source: ONLINE INFORMATION REVIEW    Volume: 33    Issue: 2    Pages: 376-
385    Published: 2009  

Times Cited: 0     References: 44     Citation Map  

Abstract: Purpose - The purpose of the paper is to explore the extent of 
the absence of data elements that are critical from the perspective of 
scientometric evaluation of the scientific productivity and impact of 
countries in terms of the most common indicators - such as the number of 
publications, the number of citations and the impact factor (the ratio of 
citations received to papers published), and the effect these may have on 
the h-index of countries - in two of the most widely used citation-enhanced 
databases.

Design/methodology/approach - The author uses the Scopus database and 
Thomson-Reuters' (earlier known as ISI) three citation databases (Science, 
Social Sciences and Arts & Humanities), both as implemented on the Dialog 
Information Services (Thomson ISI databases) and on the Web of Knowledge 
platform, known as Web of Science (WoS). The databases were searched to 
discover how many records they have for each year, how many of those have 
cited references for each year, and what percentage of the records have 
other essential or often used data elements for bibliometric/scientometric 
evaluation.

Findings - There is no difference between the databases in the presence of 
publication year data all of them include this element for all the records. 
The presence of the language field is comparable between the Thomson and 
Scopus databases, but it should be noted that a 2 per cent difference for 
mega-databases of such size is not entirely negligible. The rate of 
presence of the subject category field is better in Scopus, even though it 
has far fewer subject categories (27) than the Thomson databases (well over 
200). The rate of absence of country identification is the most critical 
and disappointing. It is caused primarily by the fact that journals have 
not had consistent policies for including the country affiliation of the 
authors. The huge 34 percent omission rate of country identification in 
Scopus also hurts its impressive author identification feature. 
Unfortunately, the country information is not available in more than 12 
million records.  

Originality/value - Irrespective of the reasons for the very high rate of 
omission of country names or codes, it should be realised and prominently 
mentioned in any scientometric country reports. The author has never seen 
this mentioned in published papers, nor in the manuscripts that he has peer 
reviewed. Many can live with the low omission rates of the language, 
document type and subject category elements, and many can just avoid using 
these filters. The two factors that define the level of distortion in the 
assessment and ranking of the research achievements of countries are the 
rate of cited reference enhanced records and the rate of presence of 
country affiliation data. 

Reprint Address: Jacso, P (reprint author), Univ Hawaii, Honolulu, HI 96822 
USA  

Addresses: 
1. Univ Hawaii, Honolulu, HI 96822 USA  

Publisher: EMERALD GROUP PUBLISHING LIMITED, HOWARD HOUSE, WAGON LANE, 
BINGLEY BD16 1WA, W YORKSHIRE, ENGLAND  
IDS Number: 447HU  
ISSN: 1468-4527  
DOI: 10.1108/14684520910951276  

CITED REFERENCES:
1.  *SCOP  SCOP DET FACTS FIG : 2009 
2.  *THOMS REUT   WEB SCI : 2009 
3.  BARILAN J Some measures for comparing citation databases 
JOURNAL OF INFORMETRICS 1 : 26 DOI 10.1016/j.joi.2006.08.001 2007 
4.  BARILAN J  Which h-index? - A comparison of WoS, Scopus and Google 
Scholar SCIENTOMETRICS 74 : 257 DOI 10.1007/s11192-008-0216-y 2008 
5.  BORNMANN L  The state of h index research Is the h index the ideal way 
to measure research performance? EMBO REPORTS 10 : 2 DOI 
1038/embor.2008.233 2009 
6.  BORNMANN L  Are there better indices for evaluation purposes than the h 
index? a comparison of nine different variants of the h index using data 
from biomedicine JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 
AND TECHNOLOGY 59 : 830 DOI 10.1002/asi.20806 2008 
7.  BORNMANN L  Convergent validity of bibliometric Google Scholar data in 
the field of chemistry-Citation counts for papers that were accepted by 
Angewandte Chemie International Edition or rejected but published 
elsewhere, using Google Scholar, Science Citation Index, Scopus, and 
Chemical Abstracts JOURNAL OF INFORMETRICS 3 : 27 
DOI 1016/j.joi.2008.11.001 2009 
8.  BUTLER L   ETHICS SCI ENV POLIT 8 : 83 2008 
9.  CRONIN B  Using the h-index to rank influential information scientists 
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY 57 : 
1275 DOI 10.1002/asi.20354 2006 
10.  CSAJBOK E  Hirsch-index for countries based on essential science 
indicators data SCIENTOMETRICS 73 : 91 DOI 10.1007/s11192-007-1859-9 2007 
11.  DALUZ MP  Institutional h-index: The performance of a new metric in 
the evaluation of Brazilian Psychiatric Post-graduation Programs 
SCIENTOMETRICS 77 : 361 DOI 10.1007/s11192-007-1964-9 2008 
12.  DEARAUJO AFP  Increasing discrepancy between absolute and effective 
indexes of research output in a Brazilian academic department 
SCIENTOMETRICS 74 : 425 DOI 10.1007/s11192-007-1817-6 2008 
13.  DEMOYAANEGON F  Coverage analysis of Scopus: A journal metric approach 
SCIENTOMETRICS 73 : 53 DOI 10.1007/s11192-007-1681-4 2007 
14.  DESS HM  SCOPUS : 2006 
15.  FINGERMAN S  SCOPUS: Profusion and confusion ONLINE 29 : 36 2005
16.  FINGERMAN S  WEB SCI SCOPUS CURRE : 2006  
17.  GAVEL Y Web of Science and Scopus: a journal title overlap study 
ONLINE INFORMATION REVIEW 32 : 8 DOI 10.1108/14684520810865958 2008  
18.  GOODMAN CD Fatty acid biosynthesis as a drug target in apicomplexan 
parasites CURRENT DRUG TARGETS 8 : 15 2007   
19.  GORMAN GE "They can't read, but they sure can count" Flawed rules of 
the journal rankings game ONLINE INFORMATION REVIEW 32 : 705 DOI 
1108/14684520810923872 2008 
20.  HOOD WW Informetric studies using databases: Opportunities and 
challenges SCIENTOMETRICS 58 : 587 2003 
21.  JACSO P  Content evaluation of databases ANNUAL REVIEW OF INFORMATION 
SCIENCE AND TECHNOLOGY 32 : 231 1997   
22.  JACSO P  CONTENT EVALUATION T : 2001 
23.  JACSO P  As we may search - Comparison of major features of the Web of 
Science, Scopus, and Google Scholar citation-based and citation-enhanced 
databases CURRENT SCIENCE 89 : 1537 2005   
24.  JACSO P  SEARCHING FOR SKELETONS IN THE DATABASE CUPBOARD .1. ERRORS 
OF OMISSION DATABASE 16 : 38 1993   
25.  JACSO P  Comparison and analysis of the citedness scores in web of 
science And Google Scholar DIGITAL LIBRARIES: IMPLEMENTING STRATEGIES AND 
SHARING EXPERIENCES, PROCEEDINGS 3815 : 360 2005   
26.  JACSO P  Testing the calculation of a realistic h-index in Google 
Scholar, Scopus, and Web of Science for F. W. Lancaster 
LIBRARY TRENDS 56 : 784 2008   
27.  JACSO P  Savvy searching - Google Scholar revisited 
ONLINE INFORMATION REVIEW 32 : 102 DOI 10.1108/14684520810866010 2008  
28.  JACSO P  ONLINE INFORM REV 32 : 262 2008  
29.  JACSO P  The pros and cons of computing the h-index using Google 
Scholar ONLINE INFORMATION REVIEW 32 : 437 DOI 10.1108/14694520810889718 
2008   
30.  JACSO P The pros and cons of computing the h-index using Scopus 
ONLINE INFORMATION REVIEW 32 : 524 DOI 10.1108/14684520810897403 2008 
31.  JACSO P The pros and cons of computing the h-index using Web of 
Science ONLINE INFORMATION REVIEW 32 : 673 DOI 10.1108/14684520810914043 
2008   
32.  JACSO P  The dimensions of cited reference enhanced database subsets 
ONLINE INFORMATION REVIEW 31 : 694 DOI 10.1108/14684520710832360 2007   
33.  JACSO P Deflated, inflated and phantom citation counts 
ONLINE INFORMATION REVIEW 30 : 297 DOI 10.1108/14684520610675816 2006 
34.  LEYDESDORFF L  Caveats for the use of citation indicators in research 
and journal evaluations JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION 
SCIENCE AND TECHNOLOGY 59 : 278 DOI 10.1002/asi.20743 2008   
35.  MEHO LI  Impact of data sources on citation counts and rankings of LIS 
faculty: Web of science versus scopus and google scholar 
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY 58 : 
2105 DOI 10.1002/asi.20677 2007   
36.  MEHO LI P 11 INT C INT SOC S : 2007   
37.  NEUHAUS C Data sources for performing citation analysis: an overview 
JOURNAL OF DOCUMENTATION 64 : 193 DOI 10.1108/00220410810858010 2008   
38.  NEUHAUS C  The depth and breadth of Google Scholar: An empirical study 
PORTAL-LIBRARIES AND THE ACADEMY 6 : 127 2006    
39.  NORRIS M  Comparing alternatives to the Web of Science for coverage of 
the social sciences' literature JOURNAL OF INFORMETRICS 1 : 161 DOI 
10.1016/j.joi.2006.12.001 2007   
40.  ROUSSEAU R  The influence of missing publications on the Hirsch index 
JOURNAL OF INFORMETRICS 1 : 2 2007   
41.  VANCLAY JK  On the robustness of the h-index JOURNAL OF THE AMERICAN 
SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY 58 : 1547 2007   
42.  VANRAAN AFJ SCIENTOMETRICS 69 : 117 2005    
43.  WHITE B NZ LIB INFORM MANAGE 50 : 11 2006   
44.  YANG K  P 69 ANN M AM SOC IN : 43 2006 