NeurIPS 2024 Area Chair Experience

Introduction

I was invited to serve as an Area Chair for NeurIPS 2024. This is the first time I served as an Area Chair for a conference and have encountered many unexpected problems and challenges.

In this blog post, I would like to share my experience as a NeurIPS 2024 Area Chair.

NeurIPS Reviewer Experience In The Past

Serving as a NeurIPS reviewer is no easy. I have been serving as a reviewer for NeurIPS since 2021 and every year I have to review at least 6 papers from the main conference and probably a few more from the workshops and specialized tracks. Even though we could bid for papers that we are interested in reviewing, the bidding process was still very noisy because I usually only have time to bid for papers only based on its title. It’s also quite normal to get papers that I have never bid for. If I got assigned a paper that is outside my expertise, changing the assigned paper is not quite possible. I would have to make up my knowledge and review the paper as best as I can.

Usually for a paper that is in my expertise and relatively well written, it would take me roughly 6 hours to complete the review. But this rarely happened. Most of the time, the paper is full of typos, errors, and descriptions that are confusing. If that’s the case, usually it could take me up to 12 hours to complete the review. In the worst scenario, if a paper is completely out of my expertise, it could take me even more than 36 hours to complete the review, because I would have to learn a lot of related mathematics and algorithms before trying to understand the paper.

Usually the reviewer has more than a month to complete the reviews of all the assigned papers. However, because of various reasons, I usually could not start the review until two weeks before the deadline. So those two weeks were usually very intense for me. Apart from my job and sleep, I would have to spend all my free time on reviewing the papers.

I am usually very confident in my assessments and I would almost always assign 4/5 or 5/5 confidence scores in my reviews. In some extremely rare cases, because there are other reviewers reviewing the same paper, contributing a review whose confidence score is low would not hurt the paper review process too much.

NeurIPS Area Chair Invitation

In April 2024, I received an Email from the NeurIPS 2024 Program Chairs inviting me to serve as an Area Chair for the conference. At first, I hesitated to accept the invitation because:

1. I am an engineer and I only collaborate with my researcher colleagues on research projects occasionally. I don’t have time to lead a research project during my daily work.
2. I think I am not as experienced as some of my other senior researcher colleagues in research.
3. Given serving as a reviewer is already very time-consuming, I am not sure if I could handle the additional workload as an Area Chair.

Later, I consulted with a very senior researcher colleague that I have been collaborating with for a long time. He encouraged me to accept the invitation and said that it would be at least a good experience for me to try it once. I also consulted with my VP and he also supported me. So I accepted the invitation.

NeurIPS Area Chair Tasks

Paper Assignment and Reviewer Assignment

Area Chair cannot bid for papers. I got assigned a batch of 13 papers most of which are in my expertise of accelerated computing.

Each paper was automatically assigned 4 reviewers. I can adjust the reviewer assignments and invite additional reviewers if needed. It’s a little bit difficult to know whether the paper was assigned to the right experts for review without at least understanding the paper abstract and the reviewer’s publication history. It turned out that the NeurIPS AI algorithm did a pretty good job in assigning right reviewers to the papers.

First Round Review

In the first round review, I wanted to review all the papers myself so that I could have a better understanding of the papers that I will finally recommend for acceptance or rejection. But it turned out that I was only able to review 6 to 7 out of the 13 papers in this period of time. As an Area Chair, I could not formally submit a review as a reviewer for the paper. What I did was I saved my notes and comments in a paper comment that’s only visible to me, Senior Area Chairs, and Program Chairs.

Emergency Review

After the first round review, unfortunately, many reviewers did not submit their reviews on time. Even though I sent out reminders to the reviewers, some reviewers never submitted their reviews. Because each paper requires at least 3 high-quality reviews, I had to invite additional emergency reviewers to review the papers that have less than 3 reviews or low-quality reviews. This was, however, a very difficult process, because I would say 9 out of 10 invited reviewers would decline the emergency review invitation.

I was fortunate to have at least 3 reviews for each paper in the end. But I feel among the one or two papers that have 3 reviews, not all the reviews were of high quality.

Review Disclosures

Before the reviews were released to the paper authors, I had opportunities to read through most of the reviews and ask for clarification or adjustments from the reviewers if I think it’s necessary. Some reviewers would reply to my comments and make adjustments to their reviews. Some reviewers would not do anything as if they have not received my requests. In fact, those reviewers would not interact with the authors either in the later reviewer - author discussion phase.

Author Rebuttal and Reviewer - Author Discussion

Even though Area Chair is supposed to drive the discussion between the reviewers and the authors, I found it very difficult. A reviewer who did not respond to the rebuttals would not respond to the Area Chair either. Consequently, there are a few papers in which the rebuttals from the authors got no further feedback or rate adjustments from the reviewers.

Reviewer - Author Discussion and Metareview

Writing meta-review could be very difficult.

When I was a reviewer previously, I did not feel so much responsibility because I could “safely” assume that the Area Chair and all the reviewers could collectively contribute to a fair meta-review, even if the reviews from a few reviewers had mistakes or low confidence and quality. But now serving as an Area Chair, I felt I had to get my meta-review absolutely right.

For the papers that have very low scores from most of the reviewers, it’s usually very straightforward to write a meta-review. But for the papers that have mixed reviews or medium scores, it’s very difficult to consolidate the reviews and decide whether to recommend the paper for acceptance or not. Even for the highly rated papers, to recommend the paper for acceptance, I would have to ensure that the paper is indeed of high quality and the reviewers did not miss any critical points.

Therefore, even though I know it might be inefficient, I ended up carefully reading all the 13 papers I was assigned. It turned out that some highly rated papers were written in a way to fool the reviewers and they succeeded. I was able to capture those oversights, discuss with the reviewers if necessary in the last minute, and decide not to recommend those paper for acceptance.

For the papers that have high scores but no recommendation for acceptance, I had an option to discuss with the Senior Area Chairs before they make the final decision.

Area Chair - Senior Area Chair Discussion

I did want to discuss with the Senior Area Chairs about one or two papers that I wanted to decline, at least to get some agreement from the Senior Area Chairs. But it turned out that those discussions never happened and my decisions became final.

References

• NeurIPS Area Chair Guidelines