Aksnes D. "Citations and their use as indicators in science policy. Studies of validity and applicability issues with a particular focus on highly cited papers" - Ph.D. Dissertation - March 2005

[Eugene Garfield]

Dag Aksnes : e-mail : dag.w.aksnes at nifustep.no


In March 2005 Dag W. Aksnes from Norway obtained his PhD degree at the
University of Twente, The Netherlands. His dissertation, Citations and
their use as indicators in science policy. Studies of validity and
applicability issues with a particular focus on highly cited papers, can
now be downloaded from the following web page:


http://english.nifustep.no/norsk/publikasjoner/citations_and_their_use_as_in
dicators_in_science_policy


The dissertation aims at contributing towards the discussion on the use of
citations as indicators. Based on different part-projects the thesis
explores an intersection between the bibliometric and sociological
questions about the phenomenon of citations, and the science policy issues
about using such data as indicators in decision-making and evaluation. A
particular focus is directed towards highly cited papers. Because citation
distributions are extremely skewed in which most publications are poorly
cited or not cited at all and a few publications are very highly cited, it
is clear that this phenomenon has to be taken into account when
constructing and using citation indicators. It is thus of particular
interest to analyse patterns of highly cited papers. This then leads on to
a number of studies of the methodological basis and validity of citation
indicators. Among the issues addressed are field-delineation, self-
citations and scientists’ perceptions of citations and citation indicators.
The dissertation consists of a collection of seven journal articles. The
core of articles is preceded by two introductory chapters.

The dissertation also contains reprints of several of the author's papers
from the journals Scientometrics, Research Evaluation and JASIST


References
Abt, H. A. (2000). Do important papers produce high citation counts?
Scientometrics, 48(1), 65-70.

Amsterdamska, O., & Leydesdorff, L. (1989). Citations: Indicators of
Significance? Scientometrics, 15(5-6), 449-471.

Bonzi, S., & Snyder, H. W. (1990). Patterns of self citation across fields
of inquiry. Proceedings of the ASIS annual meeting, 27, 204-207.

Bourke, P., & Butler, L. (1996). Publication types, citation rates and
evaluation. Scientometrics, 37(3), 437-494.

Braun, T., Glänzel, W., & Schubert, A. (1989). Assessing assessments of
British science - some facts and figures to accept or decline.
Scientometrics, 15(3-4), 165-170.

Braun, T., Glänzel, W., & Grupp, H. (1995). The scientometric weight of 50
nations in 27 science areas, 1989-1993. Part 1. All fields combined,
mathematics, engineering, chemistry and physics. Scientometrics, 33(3), 263-
293.

Butler, L. (2003). Modifying publication practices in response to funding
formulas. Research Evaluation, 12(1), 39-46.

Callon, M., Law, J., & Rip, A. (Eds.). (1986). Mapping the Dynamics of
Science and Technology. London: Macmillan.

Campanario, J. M. (1993). Consolation for the Scientist: Sometimes it is
Hard to Publish Papers that are Later Highly-Cited. Social Studies of
Science, 23, 342-362.

Campanario, J. M. (1996). Using Citation classics to study the incidence of
serendipity in scientific discovery. Scientometrics, 37(1), 3-24.

Cano, V., & Lind, N. C. (1994). Citation Life Cycles of Ten Citation
Classics. Scientometrics, 22(2), 297-312.

Case, D. O., & Higgins, G. M. (2000). How Can We Investigate Citation
Behavior? A Study of Reasons for Citing Literature in Communication.
Journal of the American Society for Information Science, 51(7), 635-645.

Chubin, D. E., & Hackett, E. J. (1990). Peerless Science. Peer Review and
U.S. Science Policy. Albany: State University of New York Press.

Cole, F. J., & Eales, N. B. (1917). The history of comparative anatomy.
Science Progress, XI, 578-596.

Cole, J. R., & Cole, S. (1972). The Ortega Hypothesis. Science, 178, 368-
375.

Cole, J. R., & Cole, S. (1973). Social Stratification in Science. Chicago:
The University of Chicago Press.

Cole, S. (1970). Professional Standing and the Reception of Scientific
Discoveries. American Journal of Sociology, 76, 286-306.

Cole, S. (1992). Making Science. Between Nature and Society. London:
Harvard University Press.

Cole, S. (2000). The Role of Journals in the Growth of Scientific
Knowledge. In B. Cronin & H. B. Atkins (Eds.), The Web of Knowledge. A
Festschrift in Honor of Eugene Garfield. Medford: ASIS.

Collins, R. (1975) Conflict Sociology. Toward an Explanatory Science. New
York: Academic Press.

Cozzens, S. E. (1989). What do citations count? The rhetoric-first model.
Scientometrics, 15(5-6), 437-447.

Cozzens, S. E. (1997). The Discovery of Growth: Statistical Glimpses of
Twentieth-Century Science. In J. Krige & D. Pestre (Eds.), Science in the
Twentieth Century (pp. 127-142). Amsterdam: Harwood Academic Publishers.

Cronin, B. (1984). The Citation Process. The Role and Significance of
Citations in Scientific Communication. London: Taylor Graham.

Davenport, E., & Cronin, B. (2000). The Citation Network as a Prototype for
Representing Trust In Virtual Environments. In B. Cronin & H. B. Atkins
(Eds.), The Web of Knowledge. A Festschrift in Honor of Eugene Garfield.
Medford: ASIS.

Egghe, L., & Rousseau, R. (1990). Introduction to Informetrics.
Quantitative Methods in Library, Documentation and Information Science.
Amsterdam: Elsevier Science Publishers.

Elkana, Y., Lederberg J., Merton, R. K., Thackray A., Zuckerman, H. (Eds.)
(1978). Toward a Metric of Science. The Advent of Science Indicators. New
York: John Wiley & Sons.

European Commission. (2001). Key Figures 2001. Special edition. Indicators
for benchmarking of national research policies. Brussels.

Garfield, E. (1977). Can Citation Indexing Be Automated? In Essay of an
Information Scientist, vol. 1 (Vol. 1). Philadelphia: ISI Press.

Garfield, E. (1979a). Citation Indexing - Its Theory and Application in
Science, Technology and Humanities. New York: John Wiley & Sons.

Garfield, E. (1979b). Is citation analysis a legitimate evaluation tool?
Scientometrics, 1(4), 359-375.

Garfield, E. (1980). The Number of Biochemical Articles Is Growing, But Why
Also the Number of References per Article. Current Contents(11), 5-9.

Garfield, E. (1990). The most-cited papers of all times, SCI 1945-1988. The
SCI top 100 - Will the Lowry method ever be obliterated. Current Contents
(7), 3-15.

Garfield, E. (1997a). Dispelling a Few Common Myths about Journal Citation
Impacts. The Scientist, 11(33), 11.

Garfield, E. (1997b). Validation of Citation Analysis. Journal of the
American Society for Information Science, 48(10), 962-963.

Garfield, E. (1998a). From Citation Indexes to Informetrics: Is the Tail
Now Wagging the Dog? Libri, 48, 67-80.

Garfield, E. (1998b). The use of journal impact factors and citation
analysis for evaluation of science. URL:
http://www.garfield.library.upenn.edu/papers/eva_of_science_oslo.html

Garfield, E., & Welljams-Dorof, A. (1992). Of nobel class: A citation
perspective on high impact research authors. Theoretical Medicine, 13(2),
117-135.

Gilbert, N. G. (1977). Referencing as Persuasion. Social Studies of
Science, 7, 113-122.

Glänzel, W., & Czerwon, H.-J. (1992). What are highly cited publications? A
method applied to German scientific papers, 1980-1989. Research Evaluation,
2(3), 135-141.

Glänzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric
research. Scientometrics, 53(2), 171-193.

Glänzel, W., Rinia, E. J., & Brocken, M. G. M. (1995). A bibliometric study
of highly cited European physics papers in the 80s. Research Evaluation, 5
(2), 113-122.

Glänzel, W., & Schubert, A. (1992). Some Facts and Figures on Highly Cited
Papers in the Sciences, 1981-1985. Scientometrics, 25(3), 373-380.

Gross, P. L. K., & Gross, E. M. (1927). College libraries and chemical
education. Science, LXVI(1713), 385-389.

Hamilton, D. P. (1990). Publishing by  - and for? - the Numbers. Science,
250(4986), 1331-1332.

Hamilton, D. P. (1991). Research Papers: Who's Uncited Now? Science, 251
(4989), 25.

Hargens, L. L. (2000). Graphing Micro-Regions in the Web of Knowledge: A
Comparative Reference-Network Analysis. In B. Cronin & H. B. Atkins (Eds.),
The Web of Knowledge. A Festschrift in Honor of Eugene Garfield (pp. 497-
516). Medford: ASIS.

Hargens, L. L., & Schuman, H. (1990). Citation counts and social
comparisons: Scientists' use and evaluation of citation index data. Social
Science Research, 19(3), 205-221.

Hicks, D. (1999). The difficulty of achieving full coverage of
international social science literature and the bibliometric consequences.
Scientometrics, 44(2), 193-215.

Irvine, J., Martin, B., Peacock, T., & Turner, R. (1985). Charting the
Decline in British Science. Nature, 316(6029), 587-590.

Katz, S. J., & Hicks, D. (1998). Indicators for Systems of Innovation (No.
IDEA 12-98). Oslo: STEP-group.

Kyvik, S. (1991). Productiviy in Academia. Scientific publishing at
Norwegian universities. Oslo: Universitetsforlaget.

Law, J. (1986). The Heterogeneity of Texts. In M. Callon, J. Law & A. Rip
(Eds.), Mapping the Dynamics of Science and Technology. London: Macmillan.

Lawani, S. M. (1982). On the Heterogeneity and Classification of Author
Self-Citations. Journal of the American Society for Information Science,
33, 281-284.

Leeuwen van, T. N., Rinia, E. J., & van Raan, A. F. J. (1996). Bibliometric
Profiles of Academic Physics Research in the Netherlands. Leiden: Centre
for Science and Technology Studies. Report CWTS 96-09.

lePair, C. (1995). Formal evaluation methods: Their utility and
limitations. International Forum on Information and Documentation, 20(4),
16-24.

Lotka, A. J. (1926). The frequency distribution of scientific productivity.
Journal of the Washington Academy of Sciences, 16(12), 317-323.

Lowry, O. H., Rosebrough, N. J., Farr, A. L., & Randal, R. J. (1951).
Protein Measurement with the Folin Phenol Reagent. Journal of Biological
Chemistry, 193, 265-275.

Luukkonen, T. (1990). Citations in the rhetorical, reward, and
communication systems of science. Unpublished PhD thesis, University of
Tampere, Tampere.

Luukkonen, T. (1991). Citation indicators and peer review: their time-
scales, criteria of evaluation, and biases. Research Evaluation, 1(1), 21-
30.

Luukkonen, T. (1997a). Quantitative Techniques in Evaluation in Western
Europe. In M. S. Fankel & J. Cave (Eds.), Evaluating Science and
Scientists. An East-West Dialoge on Research Evaluation in Post-Communist
Europe. Budapest: Central European University Press.

Luukkonen, T. (1997b). Why has Latour's theory of citations been ignored by
the bibliometric community? Discussion of sociological interpretations of
citation analysis. Scientometrics, 38(1), 27-37.

MacRoberts, M. (1997). Rejoinder: Validation of Citation Analysis. Journal
of the American Society for Information Science, 48(10), 963.

MacRoberts, M. H., & MacRoberts, B. R. (1987). Testing the Ortega
Hypothesis: Facts and Artifacts. Scientometrics, 12(5-6), 293-296.

MacRoberts, M. H., & MacRoberts, B. R. (1989). Problems of Citation
Analysis: A Critical Review. Journal of the American Society for
Information Science, 40(5), 342-349.

MacRoberts, M. H., & MacRoberts, B. R. (1996). Problems of citation
analysis. Scientometrics, 36(3), 435-444.

Martin, B. R. (1996). The use of multiple indicators in the assessment of
basic research. Scientometrics, 36(3), 343-362.

Martin, B. R. (1997). Factors Affecting the Acceptance of Evaluation
Results. In M. S. Fankel & J. Cave (Eds.), Evaluating Science and
Scientists: An East-West Dialogue on Research Evaluation in Post-Communist
Europe. Budapest: Central European University Press.

Martin, B. R., & Irvine, J. (1983). Assessing basic research: Some partial
indicators of scientific progress in radio astronomy. Research Policy, 12,
61-90.

Martin, B. R., & Skea, J. E. F. (1992). Academic Research Performance
Indicators: An Assessment of the Possibilities. Brighton: University of
Sussex.

May, K. O. (1968). Growth and Quality of the Mathematical Literature. ISIS,
59,363-371

May, R. (1997). The scientific wealth of nations. Science, 275(5301), 793-
796.

Merton, R. K. (1968). The Matthew Effect in Science. Science, 159, 56-63.

Merton, R. K. (1979). Foreword. In E. Garfield (Ed.), Citation Indexing -
Its Theory and Application in Science, Technology, and Humanities. New
York: John Wiley & Sons.

Merton, R. K. (1988). The Matthew Effect in Science, II. Cumulative
Advantage and the Symbolism of Intellectual Property. ISIS, 79, 606-623.
van der Meulen, B. J. R. (1997). The use of S&T indicators in science
policy. Dutch experiences and theoretical perspectives from policy
analysis. Scientometrics, 38(1), 87-101.

Miquel, J. F., Ojasoo, T., Okubo, Y., Paul, A., & Dore, J. C. (1995). World
science in 18 disciplinary areas: Comparative evaluation of the publication
patterns of 48 countries over the period 1981-1992. Scientometrics, 33(2),
149-167.

Moed, H. F. (2002). The impact-factors debate: the ISI's uses and limits.
Nature, 415, 731-732.

Moed, H. F., Burger, W. J. M., Frankfort, J. G., & van Raan, A. F. J.
(1985). The application of bibliometric indicators: Important field- and
time-dependent factors to be considered. Scientometrics, 8(3-4), 177-203.

Moed, H. F., Burger, W. J. M., Frankfurt, J. G., & van Raan, A. F. J.
(1985). The use of bibliometric data for the measurement of university
research performance. Research Policy, 14, 131-149.

Moed, H. F., & van der Velde, J. G. M. (1993). Bibliometric Profiles of
Academic Chemistry Research in the Netherlands. Leiden: Centre for Science
and Technology Studies. Report CWTS-93-08.

Moed, H. F., Van Leeuwen, T. N., & Reedijk, J. (1999). Towards appropriate
indicators of journal impact. Scientometrics, 46(3), 575-589.

Moed, H. F., & Vriens, M. (1989). Possible inaccuracies occuring in
citation analysis. Journal of Information Science, 15, 95-107.

Narin, F., Hamilton, K. S., & Olivastro, D. (2000). The Development of
Science Indicators in the United States. In B. Cronin & H. B. Atkins
(Eds.), The Web of Knowledge. A Festschrift in Honor of Eugene Garfield.
Meford, New Jersey: Information Today, Inc.

Nederhof, A. J., Zwaan, R. A., De Bruin, R. E., & Dekker, P. J. (1989).
Assessing the usefulness of bibliometric indicators for the humanities and
the social and behavioural sciences. A comparative study. Scientometrics, 32
(1), 37-48.

Oppenheim, C. (1997). The correlation between citation counts and the 1992
research assessment exercise ratings for British research in genetics,
anatomy and archaeology. Journal of Documentation, 53(5), 477-487.

Oppenheim, C., & Renn, S. P. (1978). Highly Cited Old Papers and the
Reasons Why They Continue to be Cited. Journal of the American Society for
Information Science, 29(4), 225-231.

Persson, O. (2000). Decline of Swedish Science? (Bibliometric Notes, No3
2000). URL: http://www.umu.se/inforsk/BibliometricNotes/BN3-2000/BN3-
2000.htm.

Phelan, T. J. (1999). A compendium of issues for citation analysis.
Scientometrics, 45(1), 117-136.

Plomp, R. (1994). The Highly Cited Papers of Professors as an Indicator of
a Research Group's Scientific Performance. Scientometrics, 29(3), 377-393.

Price, D. J. d. S. (1965). Networks of Scientific Papers. Science, 149, 510-
515.

Price, D. J. d. S. (1970). Citation Measures of Hard Science, Soft Science,
Technology, and Nonscience. In C. E. Nelson & D. K. Pollock (Eds.),
Communication Among Scientists and Engineers (pp. 3-22). Lexington: D. C.
Heat & Co.

van Raan, A. F. J. (1993). Advanced bibliometric methods to assess reseach
performance and scientific development: basic principles and recent
practical applications. Research Evaluation, 3(3), 151-166.

van Raan, A. F. J. (1996). Advanced bibliometric methods as quantitative
core of peer review based evaluation and forsight exercises.
Scientometrics, 36(3), 397-420.

van Raan, A. F. J. (2000). The Pandora's Box of Citation Analysis:
Measuring Scientific Excellence -- The Last Evil? In B. Cronin & H. B.
Atkins (Eds.), The Web of Knowledge. A Festschrift in Honor of Eugene
Garfield (pp. 301-319). Medford: ASIS.

van Raan, A. F. J. (2005). Fatal attraction: Ranking of universities by
bibliometric methods. Scientometrics 62 (1), 133-143.

Rinia, E. J., van Leeuwen, T. N., van Vuren, H. G., & van Raan, A. F. J.
(1998). Comparative analysis of a set of bibliometric indicators and
central peer review criteria. Evaluation of condensed matter physics in the
Netherlands. Research Policy, 27, 95-107.

Rip, A. (1997). Qualitative conditions of scientometrics: The new
challenges. Scientometrics, 38(1), 7-26.

Schoepflin, U. (1992). Problems of Representativity in the Social Sciences
Citation Index. Paper presented at the Representation of Science and
Technology. Proceedings of the International Conference on Science and
Technology Indicators, Bielefeld, Germany, 10-12 June 1990.

Schubert, A., & Braun, T. (1986). Relative indicators and relational charts
for comparative assessment of publication output and citation impact.
Scientometrics, 9(5-6), 281-291.

Schubert, A., & Braun, T. (1993). Reference standards for citation based
assessments. Scientometrics, 26(1), 21-35.

Schubert, A., & Braun, T. (1996). Cross-field normalization of
scientometric indicators. Scientometrics, 36(3), 311-324.

Schubert, A., Glänzel, W., & Braun, T. (1988). Against Absolute Methods:
Relative Scientometric Indicators and Relational Charts as Evaluation
Tools. In A. F. J. Van Raan (Ed.), Handbook of Quantitative Studies of
Science and Technology. Amsterdam: Elsevier.

Schwartz, C. A. (1997). The Rise and Fall of Uncitedness. College &
Research Libraries, 58(1), 19-29.

Seglen, P. O. (1989). From bad to worse: Evaluation by Journal Impact.
Trends in Biochemical Sciences, 14, 326-327.

Seglen, P. O. (1992). The Skewness of Science. Journal of the American
Society for Information Science, 43(9), 628-638.

Seglen, P. O. (1994). Causal Relationship between Article Citedness and
Journal Impact. Journal of the American Society for Information Science, 45
(1), 1-11.

Seglen, P. O. (1997). Citations and journal impact factors: questionable
indicators of research quality. Allergy, 52(11), 1050-1056.

Shadish, W. R., Tolliver, D., Gray, M., & Gupta, S. K. S. (1995). Author
Judgements about Works They Cite: Three Studies from Psychology Journals.
Social Studies of Science, 25, 477-498.

Sivertsen, G. (1992). Should a new bibliometric database for international
comparisons be more restricted in journal coverage. Paper presented at the
Science and technology in a policy context. Leiden Conference on Science &
Technology Indicators,, Leiden, The Netherlands, 23- 25 October 1991.

Sivertsen, G. (1997). Ethical and Political Aspects of Using and
Interpreting Quantitative Indicators. In M. S. Frankel & J. Cave (Eds.),
Evaluating Science and Scientists: An East-West Dialogue on Research
Evaluation in Post-Communist Europe. Budapest: Central European University
Press.

Small, H. (1982). Citation Context Analysis. In B. Dervin & M.-. Voigt
(Eds.), Progress in communication sciences (Vol. 3, pp. 287-310). Norwood:
Ablex.

Small, H. G. (1978). Cited Documents as Concept Symbols. Social Studies of
Science, 8, 327-340.

Smith, L. C. (1981). Citation Analysis. Library Trends, 30(1), 83-106.

Snyder, H., & Bonzi, S. (1998). Patterns of self-citation across
disciplines (1980-1989). Journal of Information Science, 24(6), 432-435.

Tagliacozzo, R. (1977). Self-citation in scientific literature. Journal of
Documentation, 33(4), 251-265.

Taxt, R. E., & Aksnes, D. W. (2003). Vitenskapelig publisering i perioden
1995-2000. En analyse av Det matematisk-naturvitenskapelige fakultetet.
Bergen: Universitetet i Bergen.

Testa, J. (1997). The ISI Database: the journal selection process. The ISI
Essays. URL: http://www.isinet.com/isi/hot/essays/199701.html.

Tijssen, R. J. W. (2003). Scoreboards of research excellence. Research
Evaluation, 12(2), 91-103.

Tijssen, R. J. W., Visser, M. S., & van Leeuwen, T. N. (2002). Benchmarking
international scientific excellence: are highly cited research papers an
appropriate frame of reference? Scientometrics, 54(3), 381-397.

Turner, S. P., & Chubin, D. E. (1976). Another appraisal of Ortega, the
Coles, and science policy: the Ecclesiastes hypothesis. Social Science
Information, 15(4/5), 657-662.

Turner, S. P., & Chubin, D. E. (1979). Chance and eminence in science:
Ecclesiastes II. Social Science Information, 18(3), 437-449.

Vinkler, P. (1986). Evaluation of some methods for the relative assessment
of scientific publications. Scientometrics, 10(3-4), 157-177.

Vinkler, P. (1998). Comparative investigation of frequency and strength of
motives toward referencing. The referencing threshold model.
Scientometrics, 43(1), 107-127.

Voos, H., & Dagaev, K. S. (1976). Are All Citations Equal? Or, Did We Op.
Cit. Your Idem? The Journal of Academic Librarianship, 1(6), 19-21.
VSNU. (1996). Quality Assessment of Research - Chemistry, past performance
and future perspectives. Utrecht: VSNU.

Warner, J. (2000). A critical review of the application of citation studies
to the Research Assessment Exercises. Journal of Information Science, 26
(6), 453-460.