Best Paper Award Chair

Best Paper Award Chairs are responsible for selecting conference awards complying with the ACL Conference Awards Policy.

This handbook describes implementation of the selection process.

Nominations

In the review process, reviewers (Note: PCs may choose whether or not to include reviewers in the nomination process.), AEs, ACs, and SACs will be asked to answer the following questions:

  1. "Could the camera-ready version of this paper merit consideration for an "outstanding paper" award (up to 2.5% of accepted papers will be recognized in this way)? Outstanding papers should be either fascinating, controversial, surprising, impressive, or potentially field-changing. Awards will be decided based on the camera-ready version of the paper."
    • "Yes"
    • "Maybe"
    • "No"
  2. "If yes/maybe, please briefly describe why:"
    • (short answer box with no word or character limit)

The best paper committee should consider any paper that was marked "Yes" or "Maybe" by any reviewer (Note: PCs may choose whether or not consider papers that were labeled 'maybe' by a reviewer and not nominated by the AE/AC.), AE, AC, or SAC (Note: This requirement is included to clarify how this policy interacts with ACL Rolling Review.).

SACs may also choose a paper in their area to receive the "Area Chair's Award". This will encourage diversity in the papers that are highlighted. SACs may not give this award to a paper that one of the SACs in the track has a conflict-of-interest (COI) on, but any other paper in their track can be chosen. Papers that SACs have COIs on can be nominated for other awards. If a conference has a COI track, that track does not select an Area Chair's award. The process for SACs to choose the award is up to the SACs (e.g., whether and how to involve ACs). The Best Paper Committee will not be told which papers have received this award, to avoid biasing their choices.

Papers are not eligible for awards if their authors include Program Chairs. The PCs may also choose to specify that authors in other senior organisational roles are not eligible for awards.

Special category awards can either be handled in the same way, by editing the question above to include them, or through a separate process defined by the PCs.

The eligibility for non-publicized paper awards should be confirmed before notification, i.e., in-between the final acceptance decision by PCs and notification. The best paper committee chairs or program chairs search for paper titles and authors of the candidates at preprint servers, popular social media, or on the web in general.

Organization of Best Paper Committee

Program chairs should select and appoint a Best Paper Committee Chair(s) early in the process (it is recommended well before the submission deadline), who will manage the entire process. A best paper committee will be selected by the program chairs together with the Best Paper Committee Chair(s). The committee size should be large enough to keep the load to around 10-15 papers per member. The committee should be diverse in composition in terms of research areas and demographics. It is best to form the committee before the review process commences, otherwise it may be hard to recruit members.

Collection of Publicized Status and Anonymized PDFs

The program chairs should collect (1) publicized/non-publicized status and (2) anonymized versions of PDFs at the time of camera-ready submission. Both are critical for the award selection process. The simplest way should be to use the camera-ready submission form and ask everyone to submit these.
If this process turns out to be difficult for some reason and the papers are not anonymized, then the committee chair is encouraged to manually redact author information on the first page of the paper.

Selection Process

The process below was developed based on the assumption that the nomination process leads to 50-80 papers being considered for awards. If the number of nominated papers turns out to be much higher, this process will not scale. Best paper award chairs have latitude and discretion for the design of the selection process. PCs should approve the process.

The committee will follow a multistage process to determine the final awards. The committee will receive the (anonymized) camera-ready version of the paper, anonymized reviews, and associated supplementary materials. If the papers are not anonymized then the committee chair is encouraged to redact author information on the first page of the paper. Also, the chair should encourage the committee to not look for revealing author information elsewhere (e.g. acceptance list on the conference page). 1. Papers are divided between the committee members based on research areas for a first pass in which each paper is read by at least 2 committee members. They independently place the papers into three groups: (1) consider for best paper, (2) consider for outstanding paper, (3) do not consider further. 2. All committee members read the papers that both readers placed under consideration for 'best paper'. 3. The committee meets to make the final selections.

This proposal does not define a specific rubric. However, while reading and discussing papers, committee members should consider:

  • Is each paper either fascinating, controversial, surprising, impressive, or potentially field-changing? Note that papers do not need to demonstrate all of these properties; any property is sufficient.
  • Does the paper present as its motivating use case an application with significant negative social impact? Even if the motivating use case is an application with neutral or positive social impact, are there obvious applications with significant negative social impact which are left unaddressed or insufficiently addressed by the paper? (In such cases, the paper should not be an award candidate.)
  • Is the work presented in the paper reproducible? For example, is there sufficient information in the paper to repeat the experiments? If not, is the lack of reproducibility justified in the paper?
  • Do the awards highlight a broad range of research types and strengths?
  • Do the awards include types of research that can be conducted at small labs?
  • Do the awards include papers that show excellence in potential positive social impact?

Special awards may be selected by the best paper committee or by separate committees. All award decisions may be overruled by the PCs.