{Seminar} @ CDS: #102 : 23rd June: “Security and Privacy in Peer Review”

Peer review is the backbone of scientific research, used to review hundreds of thousands of papers and allocate billions of dollars of grants annually. We will discuss two challenges, and proposed solutions, in peer review.

1) Security: A number of research communities are struggling with the breakdown of peer review where a group of reviewers make a pact with each other of trying to get assigned each others’ papers and then accept each others’ papers. We will review various such attacks as well as defenses proposed in the literature. We will discuss the assumptions underlying each of these defenses. We will then present a defense — based on randomized assignments of reviewers to papers — that does not require these assumptions. We will present theoretical guarantees as well as discuss practical deployments for review of thousands of papers.

2) Privacy: Anonymity of reviewers is a bedrock of peer review that enables reviewers to critique freely. However, in an analysis of a top conference with thousands of papers and reviewers, we find that the routine human habit of “batching” tasks — where people keep aside blocks of time for certain kinds of tasks — can allow authors to identify the reviewers of their papers. We then provide mechanisms that can impart anonymity despite batching, and provide strong theoretical guarantees of our mechanisms.

A survey article on challenges, experiments, and computational tools in peer review can be found here: https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf