ICML 2025 Area Chair Experience

Introduction

This year, I was invited to serve as an area chair for ICML 2025. This is my first time serving as an area chair for ICML and my second time serving as an area chair for a major conference.

It turns out that the area chair work for ICML is somewhat more difficult than that for NeurIPS, which I served as an area chair for last year. In this blog post, I would like to share my experience as an area chair for ICML 2025.

ICML 2025 Area Chair Experience

OpenReview Experience

Compared to NeurIPS 2024, the OpenReview interface for ICML 2025 is more sophisticated and consequently more confusing. Unlike NeurIPS 2024, ICML 2025 does not allow AC-reviewer-author three-party discussions, which makes the communication very cumbersome. In many situations, I had to manually copy and paste the “private” technical messages that I feel should be discussed from the reviewers to the authors and vice versa. When there were many mathematical equations in those messages, doing such copy and paste became extremely tedious. In my own opinion, the three-party discussion should always be enabled for technical discussions.

Relying on the OpenReview cache to save long comments and meta-reviews is not a good idea, because we could always lose it. I had lost one lengthy meta-reviews once with it and I had to start from scratch completely. So it is always a good idea to submit long comments and meta-reviews and revise them later if necessary.

AC-Reviewer Experience

This year, only two reviewers did not submit their reviews. What’s worth noting was that actually many reviewers did not submit their reviews on time, which made me panic after the reviewing period was over. I had to send them messages to confirm if they would submit their reviews in a few days after the due date, but at the same time, I had to start inviting emergency reviewers. In my opinion, responding to AC quickly on whether the reviewer could complete the review is very important, even if the reviewer is not able to complete the review due to some reasons. This will help the AC to decide whether inviting emergency reviewers are necessary. Of course, inviting emergency reviewers is not easy. Roughly one out of ten reviewers would accept the invitation. This year, I even had to use my personal connections to invite emergency reviewers from my colleagues at NVIDIA.

I read a few submitted papers during reviewing period and completed reading all the submitted papers after the reviews and scores were submitted from the reviewers. For those papers that are withdrawn, I did not read them. For those papers that have extremely low scores, I spent less time reading the details. For the rest of the papers, sometimes I sent messages and comments to the reviewers asking detailed technical questions to help me better understand the paper or confirm if they had understand the paper correctly. This also allowed me to catch some sloppy reviews and ask the reviewers for additional clarifications or corrections.

After the authors had submitted their rebuttals, many reviewers failed to respond to the rebuttals, even if I sent them messages asking them to respond. This made the reviewing and recommending process somewhat difficult.

AC-SAC Experience

Last year, when I was serving as an area chair for NeurIPS, I got zero communications from the senior area chairs. This time, I got messages from the senior area chairs even before the reviewing period began. Because the ICML reviewing process is more complicated and I was inexperienced, the SAC answered many of my questions via Email, as I was going through the process. I really appreciate his help and support.

When it came to making decisions, the SAC set up a meeting with me to discuss and calibrate the papers that I was hesitant to accept. The SAC asked critical technical questions about the paper and drove me through the decision-making process. Some of the technical questions are actually difficult to answer, even if I thought I have understood the paper very well.

The SAC I reported to has served ICML for over twenty years. He told me that a few years ago, the SAC and the ACs reported to him used to even fly to meet in person to discuss which papers should be accepted. During such decision making period, he could even read 200 papers in a few days, which sounds crazy to me.

Paper Reviewing and Decision Making Experience

In many papers I reviewed, I found the mathematical equations or derivations were very sloppy, which made the reading experience extremely unpleasant. There could be critical errors in mathematics, sometimes due to typos, which made me doubt the correctness of the theory or the algorithm. Sometimes, very critical derivations were skipped, which made the reviewers unable to verify the correctness of the correctness of the theory or the algorithm. Either way, I believe the authors must have written the paper in a hurry and there was hardly any proofreading before the paper was submitted. The authors should not assume the reviewers are mathematicians who could comprehend the sloppy mathematical equations immediately. If I were a reviewer, I would not give a high score to the papers which destroyed my reading experience because of the sloppy mathematics.

What’s more surprising to me was that for the papers whose mathematics I found problematic, the reviewers could hardly find the problems. Some of the reviewers were even commenting that “the paper is well written and easy to follow”. Apparently, the reviewers did not read the paper carefully enough to catch the problems.

In addition, I also found that some papers were written in a way that made the reviewers think the papers were much more novel than they actually were. For example, the “meta-algorithms” proposed in some papers did not cite the algorithmic innovations from the previous publications that inspired them. Sometimes, even if those publications were cited, they were not explicitly discussed or mentioned in the text. I don’t know if it was intentional or unintentional, but those papers had succeeded in fooling the reviewers in many cases, resulting in inflated scores. Remember, we could see further, because we stand on the shoulders of giants. If the paper is really a great work, its credits will never be diminished by citing the previous works.

Finally, the SAC told me that the scores from reviewers are a very important criteria for paper acceptance. Accepting papers to a conference is a zero-sum game and there are guided acceptance rates for ACs and SACs. If the scores from the reviewers diverge too much, it is very difficult for AC to defend the decision of accepting the paper that has a few low scores, even if the AC is interested in the paper very much in person.

References

• NeurIPS 2024 Area Chair Experience