Using AI to Bring a (Poetic) Justice in the Academic Review Process
“Reviewer 2” is the label of the grumpy, extremely critical reviewer, whose reviews ruin motivation, destroy weekend moods or even careers. Reviewer 2 abuses the anonymity of the review process by using a sniper to shoot malicious remarks towards the authors from her/his dark place, feeling superior and invulnerable while doing it. This is the side effect of the anonymity which is otherwise beneficial to keep the criticism in the overall review process impersonal and prevent a chain or retaliatory actions.
Here by academic review I am referring not only to the scientific journals, but also the anonymous reviews in various research agencies or award committees. In those committees, a group of (usually senior) academics are judging the ideas and achievements of their (usually more junior) colleagues.
Yet, the key in the anonymous peer review process is that it is a peer process, such that the same person can take the roles of a reviewer, author, award applicant or award judge. Motivated by seeing and experiencing biases in the peer review, it is interesting to ponder about the following questions. How to bring more empathy to Reviewer 2? How would Reviewer 2 react if she is constantly getting grumpy and malicious reviews written in her own style? How would the award judges grade someone that looks exactly as themselves at the same career stage?
Artificial Intelligence (AI) tools can be handy for addressing these questions. AI has been already considered for use in a peer review process to provide better summary of the papers, identify works that do not meet the scientific standards, detect plagiarism, and similar. This is different from the AI functionality considered here.
First the issue of more empathy. Let us take a member Z of a committee for a research funding or granting research award to a candidate Y. There could be an AI module that, based on the past data and the CV of Z, can present to Z a hypothetical situation in which Z is also among the contenders for the research funding with one of her/his typical "Z-ideas" and let Z self-compare to Y. This is to remove the bias for putting extremely high standards to the novelty of the ideas of others, but definitely not so high to oneself. Or, in a review for an academic journal, the AI module supplies Reviewer 2 with its own, AI-generated view, how the prior work of Reviewer 2 compares to the presently reviewed work of author Y in terms of originality and extent of the contribution.
Now the poetic justice part. Say that Z had acted as a Reviewer 2 many times, thrashing the research of others with her/his nasty reviews. Now Z writes a paper and sends it for review and the reviewer of his paper is X. Let us say that X is not a Reviewer 2-type, she is often trying to be helpful, while critical, and she provides constructive remarks. However, the review system has an AI module that has learned from the past reviews of Z and knows how Z would write this review. Now X puts her review in the system and the AI module comes up with suggestions how to reformulate it, so that it looks like the reviews that Z usually writes! For example, X writes:
"The contribution is interesting, but somewhat limited, since the set of results does not take into account the property A and it would be interesting to evaluate it, setting the parameters to value B."
and the AI module suggests to X to write instead:
"The paper has no contribution whatsoever. The author seems to either not be aware about the property A or he is intentionally avoiding to evaluate it. My 2c tip to the author is to really reconsider his career path: his incompetence is apparent from not setting the parameters to value B, a fact well known by anyone that has ever read a paper in this area."
A suitable name for this latter AI module would be TarantAIno, due to Tarantino’s movies that deal with poetic justice.
