Australia's RQF (fwd)

[Subbiah Arunachalam]

Thanks very much Linda for your clarification. In
India the office of the Principal Scientific Adviser
to the Cabinet is looking at ways to evaluate quality
and impact of research which go far beyond mere
publication and citation counts of resaerch papers.
The first phase of three independent studies is
expected to be completed by 31 December 2006. 

Regards.

Arun
[Subbiah Arunachalam]



--- Linda Butler <linda.butler at ANU.EDU.AU> wrote:


---------------------------------
of Arthur Sale's points about the Australian RQF,
particularly inrelation to IRs and the way in which
panels will access submittedpublications, are
accurate.  However, his "definition" ofquality and
impact in the RQF context is seriously misleading. 
Yes,the terms are used in an unusual way, but his
attempt to paraphrase themeaning is way off.  The
definitions contained in the officialdocument are:

• the quality of original research including
itsintrinsic merit and academic impact. Academic
impact relates to the recognition of the originality
of research by peersand its impact on the
development of the same or related discipline areas
within the communityof peers; and
• the impact or use of original research outside the
peercommunity that will typically not be
reported in traditional peer reviewed literature (that
is, the extent towhich research is
successfully applied during the assessment period for
the RQF). Broaderimpact relates to the
recognition by qualified end users that
methodologically sound andrigorous research has
been successfully applied to achieve social, economic,
environmentaland/or cultural
outcomes.

Quality is NOT a solely metrics-based exercise.  It is
thepeer assessment of 4 outputs per active researcher
(as in the RAE),informed by quantitative indicators
supplied to the panel(citations, competitive grants,
ranked outputs - details of proposedmeasures are on
the DEST website in the background papers).

Impact, the most difficult to assess, is judged from
an"evidence-based statement of claims".  Obviously,
there isa lot of detail behind that statement - again,
background papers areavailable on the DEST website. 
It will definitely not be judged inthe way outlined
below.

Linda Butler
Research Evaluation and Policy Project
The Australian National University

At 03:34 PM 17/11/2006, you wrote:
Adminstrative info forSIGMETRICS (for example
unsubscribe):
http://web.utk.edu/~gwhitney/sigmetrics.html

---------- Forwarded message ----------
Date: Fri, 17 Nov 2006 14:44:39 +1100
From: Arthur Sale <ahjs at ozemail.com.au>
To:
AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM at LISTSERVER.SIGMAXI.ORG

The Australian Government has released a definitive,
if incomplete,
description of Australia's Research Quality Framework
(RQF) which isour
equivalent of the UK's RAE. If familiar with the RAE,
you will recognizethe
family resemblance. I extract the essentials of the
RQF for aninternational
readership, and analyze some of the consequences
likely to flow from it.To
see the documentation, see
http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_i
ssues/research_quality_framework/rqf_development_2006.htm.

ESSENTIAL POINTS

1.      The first RQF assessment will be basedon
submissions by the 38
Australian universities by 30 April 2008. Funding
based on theassessment
will flow in calendar year 2009. Six years will elapse
before thenext
assessment (ie 2014), but there is provision to
shorten this.

2.      The Unit of Assessment is the ResearchGroup.
Research Groups will
be defined by up to three RFCD four-digit codes (to
allow for
multi-disciplinary groups). The RFCD classification is
uniquelyAustralian,
and for example there are six four-digit codes in the
field of ICT.
Engineering has more but for example Civil Engineering
is one. If youare
interested in the codes see
http://www.research.utas.edu.au/publications/docs/14_rfcd.doc,
thefour
digit codes are the sub-headings.

3.      Each Research Group will be allocated toand
assessed by one of 13
Panels. The Panel is determined by the primary RFCD
code. ThusMathematics,
Computing and Information Technology is Panel 4.

4.      Each University will submit an
EvidencePortfolio (EP) for each
identified Research Group. There is provision for
cross-universityResearch
Groups.

5.      The ratings will be based on Quality andImpact
separately. These
words have peculiar (ie not common-usage) meanings.
Approximately,Quality
is a bag of quantifiable metrics, and Impact is all
the soft thingslike
Fellowships of Academies, Honors, journal associate
editorships, etc.The
relative importance of Quality and Impact will vary by
Panel and is
similarly not yet resolved. Quality is based on the
best fourpublications
(Research Output) of each researcher in the group over
the six years
2002-2007, on a full list of all Research Output from
the groupincluding
honorary and emeritus professors, and on competitive
grants receivedover
the period. Impact is covered in the Context Statement
of the EP

6.      Impact for each Research Group will beassessed
on a scale of 1 (not
important) to 5 (prestigious)..

7.      Impact is rated A (outstanding) to E(poor).

8.      Research Groups which rate below 2 forQuality,
or below D for
Impact, will attract no funding to their university,
though the twofactors
are separately aggregated for the University. The
weighting of fundingis
stated to be linear with rating, but the gradient will
be determinedduring
2007.

9.      The Panels require access to theelectronic
versions of any of the
Research Output within four working days. The Panels
will (a) rankthe
outputs by things like journal impact factors, journal
standing, etc,(b)
assess citation counts, both in aggregate and by the
percentage that fallin
the top decile for the discipline, and (c) competitive
grantincome.

10.  The RQF is based on a semi-centralized IT model
(or
semi-decentralized). In other words, the full-texts of
the researchoutputs
(publications) will be held in IRs in each university,
while the RQF
secretariat will run a repository with all the EPs and
develop thecitation
counts independent of the universities (in conjunction
with Thomson
Scientific and possibly EndNote Web). The Australian
Government willbe
approached for funds to universities to establish
these IRs.

ANALYSIS FOR OPEN ACCESS

*        The RQF will actually usecitation metrics in
the assessment, not
just test them as a "shadow exercise" as in the next
RAE. Thiswill mean
that the OA citation advantage will suddenly look very
attractive to
Australian universities, though it is a bit late to do
anything aboutit
five years into a six-year window. However, with 2014
in mind, there willbe
pressure to increase citations.

*        Every university will have tohave an IR to
hold the full-text of
Research Outputs. About half already do, with EPrints
and DSpace beingthe
most popular software with a few Fedora-based
repositories andoutsourced
ProQuest hosts. There will be funding to establish
repositories.

*        I expect a mad scramble inthe smaller
universities, with
outsourcing and hosting solutions being very
attractive. Money fixes
everything. The ones that have been dithering will
regret it.

*        All Research Output generatedby all Research
Groups will have to
be in the IRs for the RQF. This may amount to 50% of
the universityresearch
production over six years, or more or less depending
on how research
intensive it is. There are two corollaries: (a) this
is Mandate byMoney,
and (b) there will be frantic activity over 2007 to
put in the backlogof
2002-2006 publications.

*        Since one does not know whatResearch Output
will be needed in
2014, and only a general clue in 2007, 100%
institutional mandatesare
likely to spring up all over the place, in the form of
Mandate by
Administration. What I mean by this is that the
deposition of the paperwill
be integrated with the already present administrative
annual requirementto
report the publication to the Australian Government.

*        Although it is nowhere statedexplicitly that
I can see, I read
between the lines that the RQF may be expecting to get
access to the
publisher's pdf. This means that it will have to be in
the repositoryas
"restricted access" in most cases or as a link to an
OA source.There is no
reason why the OA postprint cannot be there as "open
access" aswell, of
course, and if a citation advantage is to be got, it
will need tobe.

Please feel free to blog this or forward this to
anyone you think maybe
interested. My apologies for cross-posting.

Arthur Sale
Professor of Computing (Research)
University of Tasmania 
Linda Butler 
Research Evaluation and Policy Project 
Research School of Social Sciences
The Australian National University 
ACT 0200  Australia 
Tel: 61 2 61252154    Fax: 61 2 61259767 
http://repp.anu.edu.au




Send instant messages to your online friends http://uk.messenger.yahoo.com