Defining Open Peer Review: Part Three - A Community Endorsed Definition

 

ABSTRACT: This is the last of a series of posts describing OpenAIRE’s work to find a community-endorsed definition of “open peer review” (OPR), its features and implementations. As described in Parts One and Two, OpenAIRE collected 122 definitions of “open review” or “open peer review” from the scientific literature. Iterative analysis of these definitions resulted in the identification of seven distinct OPR traits at work in various combinations amongst these definitions:

• Open identities: Authors and reviewers are aware of each other’s identity.
• Open reports: Review reports are published alongside the relevant article.
• Open participation: The wider community to able to contribute to the review process.
• Open interaction: Direct reciprocal discussion between author(s) and reviewers, and/or between reviewers, is allowed and encouraged.
• Open pre-review manuscripts: Manuscripts are made immediately available (e.g., via pre-print servers like ArXiv) in advance of any formal peer review procedures.
• Open final-version commenting: Review or commenting on final “version of record” publications
• Open platforms: Review is de-coupled from publishing in that it is facilitated by a different organizational entity than the venue of publication.

In this final post we will describe the distribution and possible configurations of these traits and give a final, community-endorsed definition of OPR.

NOTE: The data for these definitions is available here. Readers are encouraged to review the data itself – perhaps there are definitions we’ve missed – or definitions you think have been coded wrongly? If so, please let us know by commenting directly in the spreadsheet or using the blog comments below!

Distribution and configurations of traits

The number of definitions over time shows a clear upward trend, with the most definitions in a single year coming in 2015. The distribution shows that except for some outlying definitions in the early 1980s, the phrase “open peer review” did not really enter academic discussion until the early 1990s. At that time, the phrase seems to have been used largely to refer to non-blinded review (i.e., open identities). We then see a big upswing from the early-mid 2000s onwards, which perhaps correlates with the rise of the rise of the openness agenda (especially open access, but also open data and open science more generally) over that period.

[caption id="attachment_1467" align="aligncenter" width="722"] Figure 1. Definitions by year


Most of the definitions 77.9% (n=95) come from peer-reviewed journal articles, with the second largest sources being books and blog posts. Other sources include letters to journals, news items, community reports and glossaries.

[caption id="attachment_1469" align="aligncenter" width="496"] Figure 2. Source-type of definition


The majority of definitions (51.6%) were identified to be primarily concerned with peer-review of STEM-subject material, while 10.7% targeted SSH material and the remainder (37.7%) were interdisciplinary.

[caption id="attachment_1472" align="aligncenter" width="488"] Figure 3. Disciplinary focus of definition


Regarding the target of peer-review, most definitions dealt with peer review of journal articles (80.7%), with the second largest group not specifying a target (16%). A small number of articles also dealt with review of data, conference papers and grant proposals.

[caption id="attachment_1473" align="aligncenter" width="502"] Figure 4. Review target type of definition


Of the 122 definitions identified, 68% (n=83) were explicitly stated, 37.7% (n=46) implicitly stated, and 5.7% (n=7) contained both explicit and implicit information.

The core traits are easily identified, as just three cover more than 99% of all definitions: Open identities combined with open reports cover 116 (95.1%) of all records. Adding open participations leads to a coverage of 121 (99.2%) records overall. As can be seen in the figure below, open identities is by far the most prevalent trait, present in 90.1% (n=110) of definitions. Open reports is also present in the majority of definitions (59.0%, n=72), while open participation is part of around a third. Open pre-review manuscripts (23.8%, n=29) and open interaction (20.5%, n=25) are also a fairly prevalent part of definitions. The outliers are open final version commenting (4.9%) and open platforms (1.6%).

[caption id="attachment_1475" align="aligncenter" width="673"] Figure 5. Distribution of traits


The various ways these traits are configured within definitions can be seen in the below Figure. Quantifying definitions in this way allows us to accurately portray exactly how ambiguously the phrase “open peer review” has been used thus far. For the literature offers a total of 22 distinct configurations of seven traits, effectively meaning that there are 22 different definitions of OPR in the literature.

A “power law” distribution can be observed in the distribution of these traits, with the most popular configuration (open identities) accounting for one third (33.6%, n=41) and the second-most popular configuration (open identities, open reports) accounting for almost a quarter (23.8%, n=29) of all definitions. There then follows a “long-tail” of less-frequently found configurations, with more than half of all configurations being unique to a single definition.

[caption id="attachment_1476" align="aligncenter" width="468"] Figure 6. Unique configurations of OPR traits ordered by prevelance in definitions in the literature

Towards a unified, community-endorsed definition of OPR

Based on this analysis, we constructed the following (provisional) definition of “open peer review”:

An umbrella term describing a variety of innovations which “open up” the traditional peer review process by modifying one or more aspects to make it more inclusive, transparent and/or accountable. Primary aspects are Open Identities, Open Reports and Open Participation (possession of one of these traits is usually sufficient to qualify a system as “Open Peer Review”). Secondary characteristics include Openness in Time,[i] Open Interaction, Open Platforms and others (these aspects are usually insufficient in themselves to qualify a system as “Open Peer Review”, but are often included in many definitions nonetheless).

Feedback on this provisional definition was then collected as part of an online survey of attitudes to OPR which OpenAIRE conducted from 8^th September to 7^th October 2016, and which received 3062 complete responses. The survey was open to all wishing to take part and distributed via social media, scholarly communications mailing lists, publisher newsletter and, in one case, a publisher internal mailing list (Copernicus publications).

Respondents were strongly in favour of this definition, with the overwhelming majority (84.5%, n= 2587) agreeing with it. Those who disagreed were invited to give feedback. Of these, many reported either feeling unqualified to judge (e.g., respondent 350: “I simply don't know enough to agree”) or that the matter was unimportant to them (e.g., r6079: “I just don't care about a definition. Won't spend time on it”). Within the remaining feedback, justified criticism tackled the style and language used:

• Style: Participants advised that the definition was “[t]oo lawerly“ (r4956), “[t]oo complex a definition to provide clarity” (r9111), that it was “confusing and covers too many aspects” (r1262) and reflective of “a long process of finding a compromise” (r2896).
• Biased language: Some respondents took issue with the lack of neutrality in the language used in the definition. As one respondent (r4530) said: “A definition should be objective/unbiased. Avoid terms like "innovation", "inclusive", "transparent" and "accountable". This reads like the opinion of an open review supporter, not like a definition.”
• Imprecision: Others felt the definition, as an umbrella term covering distinct aspects, was too broad to prove meaningful, e.g., “the options … are so broad, with various impact, that the definition becomes too hollow” (r7318). Some thought the differences between the various elements, and their varying implications, were too great to warrant their grouping under a common name (r3955). As one respondent put it “Your survey has (correctly in my view) separated out different concepts…. Lumping them together doesn't help” (r7975).
• Disagreement on core aspects of OPR: Others disagreed over which aspects should be included in the definition, e.g., “[t]he following is enough: "Open Identities, Open Reports and Open Participation". Don't make it more complicated!!!” (r4325). Others took issue with distinct aspects. Interestingly, by far the most common trait respondents felt should be excluded was open identities (e.g., “I do not think that Open Identities is Open Peer Review as it doesn't "open up" the peer review process to the reader” (r1941)) – even though, as we have seen, this is by far the most common element in existing definitions of OPR. Others meanwhile, felt that OPR should be reserved for just one specific trait such as open participation, open reports or (most commonly) open identities, e.g., “For me, the term ‘open peer review’ means ‘open identity peer review’, but not ‘open reports’, ‘open participation’ and so on” (r5243).

Based on this feedback, the definition was revised to make it shorter, more concrete, less “lawyerly”, more netural and better specify its terms. However, given the very broad ways in which the term OPR is used and the need to find a unifying definition, the objections that the term OPR should be reserved for one specific trait and that an inclusive definition was too broad to be meaningful were both rejected. If we resort to just arguing that OPR identifies one particular trait then we become just another voice adding to the ambiguity of the term. Instead, we offer a unifying, community-endorsed definition which makes clear the essential ambiguity of the term and invites users to be clear about which aspects of OPR (open identities, open reports, open participation, etc.) are under discussion in each case.

OPR definition:
Open peer review is an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the ethos of Open Science, including making reviewer and author identities open, publishing review reports and enabling greater participation in the peer review process.

The full list of traits are:

• Open identities: Authors and reviewers are aware of each other’s identity.
• Open reports: Review reports are published alongside the relevant article.
• Open participation: The wider community to able to contribute to the review process.
• Open interaction: Direct reciprocal discussion between author(s) and reviewers, and/or between reviewers, is allowed and encouraged.
• Open pre-review manuscripts: Manuscripts are made immediately available (e.g., via pre-print servers like ArXiv) in advance of any formal peer review procedures.
• Open final-version commenting: Review or commenting on final “version of record” publications
• Open platforms: Review is de-coupled from publishing in that it is facilitated by a different organizational entity than the venue of publication.

Conclusion

We have seen that the term “open peer review” is contested ground. Our aim here has been to provide some clarity as to what is being referred to when we use this term. By analyzing 122 separate definitions from the literature we have identified seven different traits of OPR which all aim at resolving differing peer review problems. Amongst our corpus of definitions we have identified 22 unique configurations of these traits – literally 22 distinct definitions of OPR in the literature. Given such a contested concept, in our view the only sensible way forward is to acknowledge the ambiguity of this term, accepting that it is used as an umbrella concept for a diverse array of peer review innovations. The theme that unifies these diverse traits is Open Science. Factors like opening identities, reports and participation all bespeak the ethos of Open Science in trying, in their differing and overlapping ways, to bring greater transparency, accountability, inclusivity and flexibility to the restricted traditional model of peer review.

Notes

[i] “Openness in Time” was presented as a unifying concept for the traits “open pre-review manuscripts” and “open final version commenting” – the key idea being that these are both elements which disrupt the standard temporal order of peer review (submission, review, publication) . After reflection, however, and based on feedback from users, it was decided that these two concepts were insufficiently similar to group together.