OpenEdition and OpenAIRE - experiments in open peer review

 As part of OpenAIRE2020, OpenEdition, in association with the Couperin consortium and the environmental science journal VertigO, is carrying out experiments in the areas of open peer review and open commentary. Although the experiments are still ongoing, we are here able to share an update on progress, along with some preliminary findings.

The journal chosen for the experiments, VertigO, is a popular environmental sciences journal that receives a large number of submissions. In addition to the high number of papers that must be reviewed, the journal also receives some contributions that for reasons of format and/or language are not ready for peer review although they are of scientific interest. The OPR experiment deals with these two types of submissions separately, via open peer review and open commentary, hosting five pre-prints each. As VertigO’s blog is hosted via OpenEdition’s scientific blog platform Hypotheses, this forms a natural home for the experiments.

[caption id="attachment_662" align="aligncenter" width="596"] Open peer review: an author responds to a referee's report


Experiment 1: Open peer review. The open peer review strand of the experiment operates much as traditional review except that names, review reports and annotations are made public. Review reports are displayed as comments to the pre-print, which the blog-form of the platform allows. Referees are also able to insert comments into the text itself using the open-source plug-in Hypothes.is. Once reports and annotations are published, a conversation can start between authors and referees. Examples are available online here (or, to display the annotations and activate Hypothes.is, here). Although debate within the experiment has been overwhelmingly cordial and constructive, authors and referees can report having difficulties in finding the right tone or maintaining a real debate. For example, in one instance an author asked for their text to be removed upon receiving a constructive but negative review and did not wish to engage a debate on a scientific level to defend his point. This case is in the minority, but is perhaps indicative of broader difficulties for authors and reviewers in giving and taking criticism in the changed mediational space of open peer review.

[caption id="attachment_663" align="aligncenter" width="602"] Open peer review: author and referee discussion via the annotation tool hypothes.is


Experiment 2: Open peer commentary. The second strand of the experiment does not aim to review pre-prints but rather to assist and guide authors to improve the quality of their papers such that they are ready for the peer review process. Hence, the commentary system is open to all, with the same technical possibilities as in the open peer review branch. Commentators can post general observations as comments to the pre-print at the bottom of the page (example) and they can use Hypothes.is to submit annotations within the text (example). A major difficulty (unexpectedly so) within this branch of the experiment has been finding commentators willing to engage. The mere technical possibility of commenting on pre-prints is often not enough to get users to comment – in such processes some mediation (by editors or others) is still required to engage possible commentators.

[caption id="attachment_664" align="aligncenter" width="606"] Open peer commentary: a contributor annotates a draft


The experiment shows that OpenEdition’s research blog platform Hypotheses, although not specifically designed for the task, is a suitable space for open peer review and commentary experimentation. (Although admittedly some technical improvements, including the direct integration of an annotation tool, would make the process easier.) This fact makes clear that experimentation in open peer review is not only about technological experimentation. Indeed, open peer review and open commentary protocols cannot exist as merely technical possibilities. Without human mediation, such protocols will be unsuccessful. Human mediation remains necessary in finding commentators and referees, explaining the process, advising authors and referees when new comments are posted, escorting users through the technical aspects and helping them maintain cordiality in critical debate when emotions run high as in the case referenced above.

Open Edition’s experiments with open peer review will finish within the next couple of months and we look forward to giving you a fuller update on the outcomes of the project then.