First Reviewing Experience
I have just submitted two reviews for the ACL 2025 conference, and overall, I would say it was a positive experience. It was partially an eye-opener and also a confirmation of some problems in our current reviewing system. So far, my whole experience with reviews has been from two dimensions:
- As an author, I received a few reviews—some were informative and insightful, while others were unfair and low-effort.
- As an “evaluator,” as this is my current research topic—Peer Review Evaluation.
What’s new this time is that I got to write reviews. In the beginning, I was anxious, as this was something new to me, and I didn’t know how to approach it. But the ACL Reviewer Guidelines do a great job explaining what should and shouldn’t be done. Another burden was that this is my area of research—“How to Write Good Reviews?"—so I felt the responsibility to apply everything I have been promoting in my reviews.
Here are some tips that I learned that might help anyone new to reviewing:
You have to read the paper at least TWICE—preferably in two different sittings—so you have a fresh mind and a new perspective.
You need to differentiate between what is a limitation and what is a suggestion/comment.
Keep an open mind. You are not the owner of the work you are reviewing; you are just evaluating it. So, when the paper doesn’t do things your way or how you think it should be done (given that the method is sound), don’t be frustrated. Simply frame your idea as a possible alternative in the suggestions.
Put yourself in the shoes of the authors and be fair! If this were your draft and you had already made an effort evaluating N models, there would always be an N+1 model to evaluate (in cases where there’s no strong reason to try that model).
Reviews are addressed to both authors and Area Chairs. Know your audience and address them accordingly. Generally, authors need actionable comments and suggestions to help them improve their draft, while ACs need a summarized key points of the paper’s strengths and weaknesses. But in both cases, all your comments must be specific—you need to make it easy for anyone reading your comments to know exactly what you are addressing and where it is mentioned in the paper.
One recurring scenario that I have seen is that review comments don’t reflect the given score. If you mentioned only minor weaknesses, don’t give an overall score of 2! You must justify your score, always!
Lastly, I would like to end with a quote from the ACL Reviewer Guidelines:
Not to mention, someone out there is reviewing your papers too. The more rude or dismissive reviews there are, the more of a norm they become, and the higher the chances you will get one yourself in the future.