Our Recent Productions

Showing 1–3 of 3

1. Working Paper2026Open Access

   The Future of Peer Review: A system-wide perspective(opens in a new tab)

   

   Pinfield, Stephen; Waltman, Ludo; Kolarz, Peter; Gregory, Kathleen; Brasil, André

   This report presents the main outcomes of the PEER REVIEW project conducted by the Research on Research Institute (RoRI) from 2023 to 2026. It draws on the substantial literature on peer review as well as 39 in-depth interviews with experts from across the research system, including policymakers, funders, publishers, editors, researchers and others. Peer review is often seen as one of the cornerstones of the research system. At the same time, many researchers and other actors in the system have expressed deep concerns about dysfunctions of peer review. In response to this, much work has been done to assess and improve peer review in individual domains, notably within research funding, scholarly publishing, career advancement, and assessment of research units.However, by considering peer review in each domain in isolation, knowledge about peer review has remained fragmented, offering a weak basis for strengthening the organisation of peer review.In this report, we take a systemic perspective on peer review across domains. Rather than focusing on incremental process-level innovations in individual domains, we conclude there is a need for system-wide reform in peer review. We outline future pathways for such reform.

   DOI: 10.6084/m9.figshare.32685351Peer reviewTechnical peer review
2. Working Paper2026Open Access

   Questioning peer review: Understanding peer review system-wide (RoRI Working Paper No. 21)(opens in a new tab)

   

   Gregory, Kathleen; Brasil, André; Kolarz, Peter; Pinfield, Stephen; Waltman, Ludo

   Peer review is omnipresent across the research system and is typically seen as a critical means of quality assurance and decision-making. However, there is a growing consensus that peer review is under strain, owing in large part to the proliferation of peer review demands in various parts of the research system. We argue that a holistic understanding of peer review across the research system is needed to move towards robust, complementary modes of evaluation that address these concerns. We draw on the findings from expert interviews to question the why (aims and purposes), who (actors), how (procedures and criteria), and what (objects) of peer review across “domains” where peer review is organised (research funding, scholarly publishing, career advancement, and research units). We then discuss the relationships between these domains, including the extent to which it may or may not be possible to combine reviewing efforts. We further identify five cross-domain themes through an inductive analysis: quality, capacity, diversity, openness, and trust. These themes interact with four contextual factors: disciplines, geographies, technology and governance. We argue that addressing these themes and their associated challenges in a holistic way is necessary to help peer review achieve its aims across the research system. In an associated project report, we draw on the understandings developed here to propose strategic pathways for working towards system-wide change.

   DOI: 10.6084/m9.figshare.32685546.v3Peer reviewTechnical peer review
3. Working Paper2026Open Access

   Opening the Black Box of Grant Peer Review: Exploratory Analysis of Linguistic and Sentiment Signals in Funding Outcomes(opens in a new tab)

   

   Bin, Adriana; Pinto, Daniela Maciel; Pereira, Cesar A. Guimarães; Cristofoletti, Evandro Coggo; Mena-Chalco, Jesus Paschoal; Salles Filho, Sérgio; Sugimoto, Cassidy R.

   Peer review plays a central role in research funding decisions, yet the evaluative language used in review reports remains insufficiently explored, particularly in funding agencies. This study examines how linguistic and sentiment characteristics of peer-review reports relate to funding outcomes at the São Paulo Research Foundation (FAPESP), Brazil, focusing on the Regular Research Grant (RRG) and Young Investigator Grant (YIG) schemes between 2000 and 2017. Drawing on 19,333 proposals and their corresponding review reports, we analyze three aggregated corpora — overall reports, proposal-related reports, and applicant track-record reports — using natural language processing and sentiment analysis. We compare approved and rejected proposals across fields of knowledge, reviewer gender, applicant gender, and peer-review report sections. Results show that reviewers use distinct textual and sentiment patterns when justifying approval or rejection. These patterns vary according to the section evaluated — applicant track record or technical proposal — and also differ by field of knowledge and, more locally, by reviewer–applicant gender configurations. The pattern is more regular and stable in RRG than in YIG. Overall, the findings show how evaluative language illuminates fairness, consistency, and transparency in grant peer review.

   DOI: 10.31235/osf.io/ykca7_v1Exploratory analysisSentiment analysisPeer reviewExploratory researchLinguistic analysis