How do you quantify AI review quality?
We started collecting feedback from researchers on Reviewer3 comments. Every comment in comes with a thumbs up ("Useful") or thumbs down feedback button.
In the first 500+ responses, we found that 90.6% of Reviewer3 comments have been rated useful.
What About at the Paper Level?
We wanted to understand this data at the paper-level. Within a given paper, what percent of the feedback is rated useful?
We found similar results when we looked at the ratings within a paper. Across 155 papers, the average upvote rate per paper is 88.4%.
The Distribution Tells a Stronger Story
The pie chart only shows us the average. What about the distribution by paper?
We find that the histogram is heavily right-skewed: for most papers, 90-100% of comments are rated useful.
The median also shows 100% of comments are rated useful within a paper.
What This Means
These numbers help us understand how we are doing and where we can improve. For most papers, nearly every comment is considered useful by the researcher.
We're continuing to collect feedback at the comment level and will keep reporting on these metrics as our dataset grows. If you'd like to see for yourself, upload your paper and rate the feedback.