DORA: SAN FRANCISCO DECLARATION ON RESEARCH ASSESSMENT

[Stevan Harnad]

On 2013-05-23, at 12:44 PM, ak103064 <gerhard.froehlich at JKU.AT> wrote:

> clearly highlight, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.

Of course it is absurd to use the journal impact factor as a proxy for (1) journal, (2) article or (3) author quality -- although it has to be admitted that in most fields there is a positive correlation (though a correlation that is increasingly weak, moving from (1) to (3)). 

Article and author citation counts are better, though still not enough. 

There are ways in which OA can help research assessment (generating and harvesting rich new metrics of importance, influence and impact), and, of course, maximising access and impact.

But research first has to be made OA, before OA can generate the rich new metrics.

I did not sign DORA, however, because of this piece of patent absurdity:

"...clearly highlight, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published."

Of course an article's quality is more important than metrics or journal name. And what's in a student's head is more important than the marks he gets. 

But the marks (metrics) are the way to assess what is in a student's head, when you do not have X-rays at your disposal! 

And of course the journal name matters, because the journal's track-record for peer review quality standards matters. 

Ideally, every single paper written should be read and evaluated by qualified experts (maybe even by God) every time its author's contributions are assessed. 

But in practice that simply cannot be done, realistically. And that's what peer review is supposed to have done. 

And the journal name has a track-record for peer review standards behind it.

I've become allergic to recommendations of the kind I quoted above, because they have so often been used as special pleading for new Gold OA journals, implying that they have an unfair handicap because they don't have an ISI impact factor, or don't have a high enough one -- whereas the truth is that they simply have no track record for peer review quality standards, hence there is no way to know what their quality standards are. 

(If every paper could in fact be read by a qualified expert every time it enters into an assessment, that would be a solution, but there's no way on earth that can be practically implemented, so we have to rely on metrics and track records.) 

In the Gold OA context, some people have also implied that authors should choose journals on the basis of their economic model (Gold OA) instead of their quality standards -- and that tenure committees should somehow give special compensatory weight to journals without track records, simply because they are OA. I think that is nonsense, and also does a disservice to OA (along lines similar to the way the predatory Gold OA journals do OA a disservice). 

The only advice to give to early-stage (or any-stage) investigators is to try to publish in the journal with the highest quality standards your work can meet -- and that their performance assessment committee should weight their work accordingly (if they cannot assign a qualified peer to re-do the peer review every time it is assessed!).

Stevan Harnad

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20130523/24c01012/attachment.html>