[bibliometrics literature coverage of literature and the web

[Paul Bracke]

We are setting up DLIST (http://dlist.sir.arizona.edu), Digital Library of
Information Science & Technology, initially an interdisciplinary LIS-focused
repository of both published and unpublished materials. To do this, we are
extending the
traditional eprints definition and including almost all the formats Bernie
Sloan started this thread off with. I write to SIGMETRICS and ISSI now,
because a priority collection development area is Informetrics and formats
include both published and unpublished papers + datasets.

To respond to Eugene, yes, scholars must share...but, academic scholars
rarely have time to 'report' and I suspect that the *interdisciplinary*
nature of our research also makes any reporting difficult. Another part of
the problem with standard databases is that they do not publish the datasets
(I did see one social science journal announce the other day that
have started doing this) that underlie the research reported.

Whether we like it or not personal collections are the future; so many
information-seeking behaviors studies have shown this that we shouldn't be
surprised!  Anyway, we hope that DLIST (as it grows) will provide such many
personal collection tools too. ISSI and SIGMETRICS folks are encouraged to
register as users on DLIST and deposit their published/unpublished papers
and datasets. This model of central deposits and decentralized management is
also (in our view) another feature of the
information landscape.

As part of DLIST, we offered to harvest or mirror Cybermetrics and sent
Isidro full details of DLIST in April. We hope that Isidro and the rest of
the community will respond to this initiative enthusiastically.

Best wishes,

Paul

--
Paul J. Bracke
Project Manager, DLIST(http://dlist.sir.arizona.edu)

Head of Systems and Networking
Arizona Health Sciences Library
1501 N. Campbell Ave.
P.O. Box 245079
Tucson, AZ  85724-5079

Phone: 520-626-1004
Fax:   520-626-2922
Email: paul at ahsl.arizona.edu

-----Original Message-----
From: ASIS Special Interest Group on Metrics
[mailto:SIGMETRICS at LISTSERV.UTK.EDU]On Behalf Of Isidro F. Aguillo
Sent: Thursday, July 11, 2002 5:37 AM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] [bibliometrics literature coverage of
literature and the web


Dear all:

I agree with Eugene about making public his message because part of my
answer is closely linked with the interesting debate about web citations.

The request to use SIGMET messages was done for the future ISSI website
in order to offer a "good" coverage of the recent literature and some
summaries about debates and other topics covered by SIGMET. The aim was
to reserve cybermetric/webometric references for the Cybermetrics
ejournal directory and use new ISSI website to cover "SI" citations.

 From my previous Cybermetrics experience it can be very useful to have
this kind of information "repositories" in the Web, consisting not only
of paper citations but considering other resources too: Directories of
personal pages, lists of unpublished presentations, links to databases
and software providers, access to databases or other raw data.

The compilation task can be really hard, but extending Eugene's
invitation, all of us can benefit if the lists are used to distribute
data about papers, web resources or new software. Then this information
will be uploaded to the websites for universal availability.

As I previously stated this effort is not going to increase our IF but
clearly the visibility will be greater.

Garfield, Eugene wrote:

> Isidro's  recent email message to SIGMETRICS reminded me that I have not
> heard from  anyone at ISSI concerning the further use and expansion of the
> SIGMETRICS archive. I think someone had asked whether the SIGMET
> information could be used by ISSI. To make things quite clear I would be
> delighted for any of my postings to SIGMET to be sent to the ISSI
listserv.
> But perhaps it is important to ask how much overlap there is in the two
> lists.
>
> I have not seen few contributions of the kind that I have been posting. If
> there are significant items outside the literature covered by ISI, as e.g.
> postings to web sites, why haven't SIGMET members or ISSI members been
> posting them to SIGMET?
>
> SIGMET is open and free to everyone. So if as Sloane and others have been
> suggesting, there is a significant amount of relevant information that is
> not revealed from the ISI databases, then why hasn't it been reported, at
> least for the field of bibliometrics,etc. in SIGMET?
>
> When I first began this note to Isidro I did not think it relevant for the
> entire SIGMET readership but I see now that is clearly is a matter that
all
> members of SIGMET and ISSI should take to heart. If we are to provide
truly
> comprehensive coverage of the field then it is the duty of all members to
> create suitable postings for all to share. Best wishes. Eugene Garfield
>
>
> When responding, please attach my original message
> __________________________________________________
> Eugene Garfield, PhD. email:  garfield at codex.cis.upenn.edu
> home page: www.eugenegarfield.org
> Tel: 215-243-2205 Fax 215-387-1266
> President, The Scientist LLC. www.the-scientist.com
> Chairman Emeritus, ISI www.isinet.com
> Past President, American Society for Information Science and Technology
> (ASIS&T) www.asis.org
>
>
>
>
>
>
> -----Original Message-----
> From: Isidro F. Aguillo [mailto:isidro at CINDOC.CSIC.ES]
> Sent: Wednesday, July 10, 2002 7:46 AM
> To: SIGMETRICS at LISTSERV.UTK.EDU
> Subject: [SIGMETRICS] Web citation issues
>
>
> Dear all:
>
> I arrive a bit late to this debate, but I think an important point is
> missing in several of the messages. Most of them focus on the
> comparision between links and citations and the reasons to cite and the
> objects to be cited. Certainly both wordls are different and printed
> literature and bibliographic references are superior for bibliometric
> analysis.
>
> But our traditional approach has several shortcommings as several
> authours have already pointed. From the evaluation and policy purposes
> point of view, one of the most important is that there are no enough
> readily available information for a real sciencetechnoeconometrics. The
> interlinked nature of the Web offer the possibility to discover hidden
> relationships among different websites, including not only the bonds
> that binds academic communities but those showing economic, industrial,
> social or cultural relationships. And this is not easy with traditional
> quantitative methods.
>
> Going to the personal aspects, the presence on the Web of research
> groups, professors or postgraduate students reflects a wider range of
> activities than formal publication in refereed journals, such as
> unpublished material, general public contributions, drafts for future
> papers or book chapters, slides used in conference or seminar
> presentations, support material for courses or even raw data. The
> picture you can obtain from a researcher is therefore more complete.
>
> Finally, as tax-payer citizen supporting public research activities I
> request better access to the results obtained by scientists and in this
> sense the Web reaches a wider audience than the paper based publications
> like journals or books. In fact almost all the information published on
> the Web can be recover by any Internet user worldwide.
>
> Comments ...
>
> Sloan, Bernie wrote:
>
>
>>Peter,
>>
>>For the past two years or so I have been dabbling in the collection of
>>citations and inlinks to several papers of mine. I use your definition of
>>"inlinks": "links pointing to web pages" (Bjorneborn & Ingwersen). I've
>>
> been
>
>>using both publicly available Web search engines (Google and AllTheWeb),
>>
> as
>
>>well as some commercial subscription-only databases from providers (e.g.,
>>ISI, the Gale Group, and EBSCO). I tried to be somewhat selective and did
>>some brief analysis of the items I retrieved, e.g., I did not count
>>
> inlinks
>
>>from personal bookmark lists, or references to the papers from discussion
>>group archives.
>>
>>I don't pretend that my papers are scientific papers, or that the
>>
> citations
>
>>and inlinks I have discovered are for "scientific items citing other
>>scientific items." I did my best to eliminate any marginal entries,
>>
> although
>
>>"marginal" is definitely in the eye of the beholder here. I like to think
>>
> of
>
>>what I am doing as using the Web to discover the "influence of ideas". The
>>exercise started with the Web of Science, and then broadened out to
>>
> include
>
>>the resources mentioned in the preceding paragraph. I don't think of the
>>exercise as something that replaces ISI databases such as SCI, SSI, etc.,
>>but rather something that supplements the information obtained from ISI.
>>
>>One of the most interesting findings is that this exercise uncovered an
>>international influence that I would not have been aware of using only the
>>ISI databases. More than 40% of the citations/inlinks were from non-US
>>sources. Of the citations/inlinks from non-US sources, fully 60% were
>>associated with countries with a primary language other than English.
>>
>>Right now I am working on putting together a paper reporting on my
>>
> findings,
>
>>discoveries and questions in more detail, sort of a report on a
>>work-in-progress. The biggest question I am grappling with is "So what?" I
>>have a detailed listing of citations/inlinks for several of my papers.
>>Beyond satisfying personal and academic curiosity, what can these
>>
> "personal
>
>>citation indices" be used for? Using them for evaluation springs to mind,
>>but of course "evaluation" requires comparison against some standard or
>>control group. I'm not aware of too many people who have conducted this
>>exercise at a personal level. I have something that seems impressive to
>>
> me,
>
>>in a relative sense (i.e., a more comprehensive overview of the influence
>>
> of
>
>>ideas than I might obtain from a citation index like SSI). But what can I
>>use it for?
>>
>>Bernie Sloan
>>
>>-----Original Message-----
>>From: Ingwersen Peter [mailto:PI at DB.DK]
>>Sent: Tuesday, July 09, 2002 9:52 AM
>>To: SIGMETRICS at LISTSERV.UTK.EDU
>>Subject: [SIGMETRICS] SV: [SIGMETRICS] Web citation issues
>>
>>Dear colleagues, as you may be well aware of our department was one of the
>>first to engage into webometric analyses, in particular of web impact by
>>means of inlinks. Several others around have done deep analyses of
>>webometric nature to see if, for instance, there exists a correlation
>>between citation impact and web impact/no. of inlinks (Bar Ilan) or
>>
> between
>
>>other indicators of recognition (e.g. RAE in the UK) and web impact
>>(Thelwall). Sometimes there exist such correlations, but the major problem
>>is that the web acts differently than scientific communication vehicles
>>
> (no
>
>>real conventions of linking etc.). Also: one may not equalize a web page
>>with a scientific article. So Tom Wilson´s no. of hits (to web pages) does
>>not necessarily correspond at all to "scientific items citing other
>>scientific items" as in conventional citation analyses. Only the reading
>>lists found by Wilson may act like such "reference lists". Further:
>>Different search engines commonly perform differently on the same search
>>profile, resulting in biased counts; several large quality studies have
>>
> been
>
>>done on "scientific" web page search results to observe the actual
>>proportion of scientific output (see e.g. Allen et al in Science in 1999.)
>>Results are appalling and pauvre - and highly dependent on the domain in
>>question. For instance, in politically hot scientific topics (like in the
>>environmental sc.-) there is a chaotic mix of scientific, semi-scientific,
>>pseudo, popular and, foremost, political opinion papers. Should all the
>>"published" pages count or only the peer reviewed ones - e.g. those
>>published by scientific institutions or referring to peer reviewed
>>
> journals
>
>>or published in peer reviewed e-journals?
>>Additionally, there are several possible web impacts: by inlink counts; by
>>inlink counts and outlink counts; by web-based traditional
>>references/citations on all the open web - or only in e-journals - with or
>>without peer review.
>>Tom will of course also run into the problem similar to that of ISI: in
>>
> the
>
>>citation databases no citations from books and non-ISI journals are
>>
> counted;
>
>>on the web only the open weblinks (and citations/references) are possible
>>
> to
>
>>count - not links/citations provided by pages/items on the hidden web
>>
> (e.g.
>
>>all the Dialog or ISI databases or the journals in publisher archives or
>>
> in
>
>>Digital Libraries.
>>Finally, on the web the nature of obsolescence of information is quite
>>different and not yet well understood - see e.g. Ronald Rousseau´s
>>
> articles
>
>>in Cybermetrics or recent publications by Wolfgang Glanzel on the issue.
>>A recent review on some of the issues touched upon above is: Bjorneborn,
>>
> L.
>
>>& Ingwersen, P.: Perspectives of webometrics. Scientometrics, 50(1):
>>
> 65-82.
>
>>Many regards - Peter Ingwersen
>>********************************************
>>Peter Ingwersen, Professor, Ph.D.
>>Department of Information Studies
>>Royal School of Library and Information Science,
>>Birketinget 6, DK 2300 Copenhagen S - Denmark
>>Tel: +45 32 58 60 66; FAX: +45 32 84 02 01
>>http://www.db.dk/pi/ - e-mail: pi at db.dk
>>Visiting Professor (Docent), Dept. of Information Studies
>>Åbo University Akademi - Finland
>>********************************************
>>
>>
>>-----Oprindelig meddelelse-----
>>Fra: Quentin L. Burrell [mailto:quentinburrell at MANX.NET]
>>Sendt: 21. juni 2002 18:11
>>Til: SIGMETRICS at LISTSERV.UTK.EDU
>>Emne: Re: [SIGMETRICS] Web citation (fwd)
>>
>>
>>I am wholly in support of the SIGMETRICS site being one for discussion and
>>so was interested in Tom's original submission and now Gene's response.
>>
> Here
>
>>comes my two penn'orth (Eng., coll., obs.?)
>>
>>Tom's observations are interesting - to his comment on token citations I
>>would add (many cases of) self-citation - but I go along with Gene's
>>uneasiness on the current haphazard coverage of the web being adequate to
>>replace formal citation bases.
>>
>>Gene's final remark that the "ultimate objective of universal
>>bibliographical control is to find it all in one place, displayed in a
>>fashion that is easily and quickly comprehended" surely requires some
>>response.
>>
>>(i)I guess that "control" was a hasty first attempt and that "information"
>>is more in line with the intended meaning.
>>
>>(ii) I would really like to see the phrase "freely available" inserted
>>somewhere in the remark. At the moment, unless you are the member of a
>>subscribing institution you don't have free access to this bibliographic
>>information, either to "boost your ego" or to measure your impact.Citation
>>analysis - like any othe form of data analysis - requires access to the
>>data.
>>
>>Anyone else willing to chip in a cent or a yen or a euro or a ... ?
>>
>>Quentin Burrell
>>
>>
>>-----Original Message-----
>>From: ASIS Special Interest Group on Metrics
>>[mailto:SIGMETRICS at LISTSERV.UTK.EDU]On Behalf Of Garfield, Eugene
>>Sent: 19 June 2002 18:27
>>To: SIGMETRICS at LISTSERV.UTK.EDU
>>Subject: Re: [SIGMETRICS] Web citation (fwd)
>>
>>
>>It is almost six weeks since Tom Wilson posted this message. Many others
>>forwarded copies to me expecting me to respond to his challenge.
>>
>>It is always pleasant to learn that one's work has been mentioned on a
>>particular web site or that it is discussed in various courses. But while
>>they are newsworthy they havoc little real bearing on the use of citation
>>indexes to measure the impact of one's research.
>>
>>When you are quite young anything helps boost the ego, but the bottom line
>>for the researcher is whether anyone has used his or her basic ideas in
>>ongoing research. Until that day of Nirvana arrives when everything will
>>
> be
>
>>searchable on the web I am afraid web searching just won't be an adequate
>>substitute.
>>
>>If you are working in the life sciences you can find many relevant
>>
> citation
>
>>connections through such full text resources as HighWire Press, but that
>>
> is
>
>>not yet complete nor is it presented in a form that is easily used for
>>citation analysis. That day may come. Steve Lawrence's project at NEC
>>
> which
>
>>provides citation indexing in context for the computer science literature
>>illustrate what happens when you have only partial coverage.
>>
>>The ultimate objective of universal bibliographical control is to find it
>>all in one place, displayed in a fashion that is easily and quickly
>>comprehended. Gene Garfield
>>
>>When responding, please attach my original message
>>__________________________________________________
>>Eugene Garfield, PhD. email:  garfield at codex.cis.upenn.edu
>>home page: www.eugenegarfield.org
>>Tel: 215-243-2205 Fax 215-387-1266
>>President, The Scientist LLC. www.the-scientist.com
>>Chairman Emeritus, ISI www.isinet.com
>>Past President, American Society for Information Science and Technology
>>(ASIS&T) www.asis.org
>>
>>
>>
>>
>>
>>
>>-----Original Message-----
>>From: Gretchen Whitney [mailto:gwhitney at UTK.EDU]
>>Sent: Monday, May 06, 2002 10:10 AM
>>To: SIGMETRICS at LISTSERV.UTK.EDU
>>Subject: [SIGMETRICS] Web citation (fwd)
>>
>>
>>---------- Forwarded message ----------
>>Date: Mon, 06 May 2002 12:25:22 +0100
>>From: Prof. Tom Wilson <t.d.wilson at SHEF.AC.UK>
>>To: JESSE at LISTSERV.UTK.EDU
>>Subject: Web citation
>>
>>There have been a few mentions of Web citation searching possibly
>>
> replacing
>
>>citation indexing in time and I wondered how many people are now, as a
>>matter of course, using counts of Web mentions in their cases for
>>appointment, tenure or promotion.
>>
>>I looked at a couple of my own papers and counted the SSCI citations and
>>then searched for mentions of the papers on the Web - the results left me
>>wondering whether the reliance on citation indexing as a measure of
>>performance is now past its sell by date.
>>
>>My most cited paper is "On user studies and information needs" (1981) - a
>>Web search (using Google) revealed 118 pages that listed the title. The
>>pages were reading lists, free electronic journals, and documents that
>>
> would
>
>>never be covered by SSCI, such as reports from various agencies. SSCI
>>revealed, if I recall aright, 79 citations of the paper. The question is:
>>
> is
>
>>the Web revealing impact more effectively than SSCI?  Citation in
>>
> scholarly
>
>>papers takes a variety of forms and much citation is of a token variety -
>>
> x
>
>>is cited because x is always cited. On the other hand citation on reading
>>lists implies some positive recommendation of the text, and mention in
>>policy documents and the like, implies (at least in some cases) that some
>>benefit has been found in the cited document.
>>
>>It may also be that the use of Web citation would provide a more complete
>>measure - I discovered, much to my surprise, that a 1971 text of mine on
>>'chain indexing' is cited on one reading list and in the bibliography of a
>>document in German on classification. Greater international coverage is a
>>further benefit of using Web citation.
>>
>>It strikes me that a move towards using Web citation as the measure of
>>performance would be rather more useful than the use of citation indexes.
>>
>>No doubt others have looked at this issue - is any consensus emerging?
>>
>>Tom Wilson
>>
>>-----------------------------------
>>Professor T.D. Wilson, PhD
>>Publisher/Editor in Chief
>>Information Research
>>University of Sheffield
>>Sheffield S10 2TN
>>United Kingdom
>>Tel: +44-114-222-2642
>>E-mail: t.d.wilson at shef.ac.uk
>>Web site: http://InformationR.net/
>>-----------------------------------
>>
>>
>>
>
>
> --
> --------------------------------------------------------
> Isidro F. Aguillo       isidro at cindoc.csic.es
> Laboratorio de Internet. CINDOC-CSIC
> Joaquin Costa, 22       Ph. +34-91-5635482 ext. 313
> 28002 Madrid            Mb. +34-630-858997
> ESPAÑA-SPAIN
>         Fx. +34-91-5642644
> Editor Cybermetrics     www.cindoc.csic.es/cybermetrics
> --------------------------------------------------------
>
>


--
--------------------------------------------------------
Isidro F. Aguillo       isidro at cindoc.csic.es
Laboratorio de Internet. CINDOC-CSIC
Joaquin Costa, 22       Ph. +34-91-5635482 ext. 313
28002 Madrid            Mb. +34-630-858997
ESPAÑA-SPAIN
        Fx. +34-91-5642644
Editor Cybermetrics     www.cindoc.csic.es/cybermetrics
--------------------------------------------------------