Australia's RQF (fwd)

[Stevan Harnad]

---------- Forwarded message ----------
Date: Fri, 17 Nov 2006 14:44:39 +1100
From: Arthur Sale <ahjs at ozemail.com.au>
To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM at LISTSERVER.SIGMAXI.ORG

The Australian Government has released a definitive, if incomplete,
description of Australia's Research Quality Framework (RQF) which is our
equivalent of the UK's RAE. If familiar with the RAE, you will recognize the
family resemblance. I extract the essentials of the RQF for an international
readership, and analyze some of the consequences likely to flow from it. To
see the documentation, see
http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_i
ssues/research_quality_framework/rqf_development_2006.htm.

ESSENTIAL POINTS

1.      The first RQF assessment will be based on submissions by the 38
Australian universities by 30 April 2008. Funding based on the assessment
will flow in calendar year 2009. Six years will elapse before the next
assessment (ie 2014), but there is provision to shorten this.

2.      The Unit of Assessment is the Research Group. Research Groups will
be defined by up to three RFCD four-digit codes (to allow for
multi-disciplinary groups). The RFCD classification is uniquely Australian,
and for example there are six four-digit codes in the field of ICT.
Engineering has more but for example Civil Engineering is one. If you are
interested in the codes see
http://www.research.utas.edu.au/publications/docs/14_rfcd.doc, the four
digit codes are the sub-headings.

3.      Each Research Group will be allocated to and assessed by one of 13
Panels. The Panel is determined by the primary RFCD code. Thus Mathematics,
Computing and Information Technology is Panel 4.

4.      Each University will submit an Evidence Portfolio (EP) for each
identified Research Group. There is provision for cross-university Research
Groups.

5.      The ratings will be based on Quality and Impact separately. These
words have peculiar (ie not common-usage) meanings. Approximately, Quality
is a bag of quantifiable metrics, and Impact is all the soft things like
Fellowships of Academies, Honors, journal associate editorships, etc. The
relative importance of Quality and Impact will vary by Panel and is
similarly not yet resolved. Quality is based on the best four publications
(Research Output) of each researcher in the group over the six years
2002-2007, on a full list of all Research Output from the group including
honorary and emeritus professors, and on competitive grants received over
the period. Impact is covered in the Context Statement of the EP

6.      Impact for each Research Group will be assessed on a scale of 1 (not
important) to 5 (prestigious)..

7.      Impact is rated A (outstanding) to E (poor).

8.      Research Groups which rate below 2 for Quality, or below D for
Impact, will attract no funding to their university, though the two factors
are separately aggregated for the University. The weighting of funding is
stated to be linear with rating, but the gradient will be determined during
2007.

9.      The Panels require access to the electronic versions of any of the
Research Output within four working days. The Panels will (a) rank the
outputs by things like journal impact factors, journal standing, etc, (b)
assess citation counts, both in aggregate and by the percentage that fall in
the top decile for the discipline, and (c) competitive grant income.

10.  The RQF is based on a semi-centralized IT model (or
semi-decentralized). In other words, the full-texts of the research outputs
(publications) will be held in IRs in each university, while the RQF
secretariat will run a repository with all the EPs and develop the citation
counts independent of the universities (in conjunction with Thomson
Scientific and possibly EndNote Web). The Australian Government will be
approached for funds to universities to establish these IRs.

ANALYSIS FOR OPEN ACCESS

*        The RQF will actually use citation metrics in the assessment, not
just test them as a "shadow exercise" as in the next RAE. This will mean
that the OA citation advantage will suddenly look very attractive to
Australian universities, though it is a bit late to do anything about it
five years into a six-year window. However, with 2014 in mind, there will be
pressure to increase citations.

*        Every university will have to have an IR to hold the full-text of
Research Outputs. About half already do, with EPrints and DSpace being the
most popular software with a few Fedora-based repositories and outsourced
ProQuest hosts. There will be funding to establish repositories.

*        I expect a mad scramble in the smaller universities, with
outsourcing and hosting solutions being very attractive. Money fixes
everything. The ones that have been dithering will regret it.

*        All Research Output generated by all Research Groups will have to
be in the IRs for the RQF. This may amount to 50% of the university research
production over six years, or more or less depending on how research
intensive it is. There are two corollaries: (a) this is Mandate by Money,
and (b) there will be frantic activity over 2007 to put in the backlog of
2002-2006 publications.

*        Since one does not know what Research Output will be needed in
2014, and only a general clue in 2007, 100% institutional mandates are
likely to spring up all over the place, in the form of Mandate by
Administration. What I mean by this is that the deposition of the paper will
be integrated with the already present administrative annual requirement to
report the publication to the Australian Government.

*        Although it is nowhere stated explicitly that I can see, I read
between the lines that the RQF may be expecting to get access to the
publisher's pdf. This means that it will have to be in the repository as
"restricted access" in most cases or as a link to an OA source. There is no
reason why the OA postprint cannot be there as "open access" as well, of
course, and if a citation advantage is to be got, it will need to be.

Please feel free to blog this or forward this to anyone you think may be
interested. My apologies for cross-posting.

Arthur Sale
Professor of Computing (Research)
University of Tasmania