COLING2020 PC-Chairs Message to Reviewers

Dear reviewers,

Thank you very much for reviewing for COLING 2020!

In addition to the usual guidelines and reminders that are useful for those that are reviewing for the first time, in what follows we introduce several recommendations on the selection criteria that, as COLING2020 PC-Chairs, we propose you to follow in paper selection. With these suggestions, we want to contribute to making COLING2020 a very special conference, as COLING ones have been since the first 1965 meeting in New York to the last edition in Santa Fe in 2018.

Please take a balanced approach when reviewing the papers. On the one hand, we would like to have a solid technical program with high-quality papers describing a complete piece of work; on the other hand, we also want a broad and interesting program, so please keep an open mind when evaluating and recommending the papers assigned to you.

Note that the Call for Papers mentions explicitly the mission of COLING to present research that is at the intersection of linguistics and computation, or, that being engineering experiments, includes linguistic insights and motivations.

Accordingly, we would like that COLING2020 selects papers with an in-depth analysis of results, going beyond the mere figures for the leaderboard and, where appropriate, provides an analysis of the results using diagnostic datasets designed to evaluate and analyze performance with respect to motivated linguistic phenomena. Sound papers describing innovative approaches and methods for working with small datasets and low-resourced languages are also very welcome. Additionally, we think that it is important to require that selected papers include a detailed description of the methodology used and the technical details that make reproducibility possible. A motivated analysis of the contribution of the paper should also be required. Also note that a short paper is not a shortened long paper. It should have a point that can be made in a few pages and present a focused contribution. Finally, we ask you to consider also the following aspects:

  • While it is easy to understand the need to address English for results comparison, the fact of addressing other languages must be considered an asset. Therefore, we encourage you to support the research on languages other than English.
  • Nevertheless, for papers addressing languages other than English, it is important to check that they are contributing with the datasets and resources that will make results comparison possible in the future.
  • NLP research is now more and more dependent on datasets that are costly to gather and it is therefore important to foster its sharing and reuse of resources. Make sure that any new dataset(s) used in a paper is well described, with complete information about how the data were gathered and annotated. Also, assess whether the dataset(s) is made easy to access and has cleared distribution rights.

Our COLING2020 review form consists of five main sections paying attention to the following aspects:

1. Summary Ranking

Please evaluate the submission according to the criteria below.

  • Relevance (1-5): Is this paper relevant to COLING?
  • Readability/clarity (1-5): Is the paper well-written and well-structured?
  • Originality (1-5): How original and innovative is the research described? Originality could be in the linguistic question being addressed, in the methodology applied to the linguistic question, or in the combination of the two.
  • Accessibility verified (NA, Yes or No): Have you personally checked the accessibility of the resource, data or code?
  • Technical correctness/soundness (1-5): Is the research described in the paper technically sound and correct? Can one trust the claims of the paper—are they supported by the analysis or experiments and are the results correctly interpreted?
  • Reproducibility (1-5): To what degree is there sufficient detail for someone in the same field to reproduce/replicate the results?
  • Substance (1-5): How much substance does this paper have? If submitted as a long-paper, is the work mature and complete or would it benefit from further development. Similarly, if submitted as a short-paper, does it address a clear point and is the research focussed?

2. Detailed Comments:

Please supply detailed comments to back up your rankings. These comments will be forwarded to the authors of the paper. The comments will help the committee decide the outcome of the paper, and will help justify this decision for the authors. Moreover, if the paper is accepted, the comments should guide the authors in making revisions for a final manuscript. Hence, the more detailed you make your comments, the more useful your review will be - both for the committee and for the authors.

Please describe what problem or question this paper addresses, as well as the main contributions that it makes towards a solution or answer. Also include the main strengths and weaknesses of this paper and the work it describes.

3. Evaluation Category

  • Overall recommendation (1-5): There are many good submissions competing for slots at COLING2020; how important is it to feature this one? Will people learn a lot by reading this paper or seeing it presented? Please be decisive—it is better to differ from other reviewers than to grade everything in the middle.
  • Confidence (1-5): How confident are you in your review, in this subject area?
  • Presentation Type (Oral or Poster): We have fewer slots for oral presentations (talks) than for posters, and want to make sure that the most appropriate papers get selected for talks. Note that the published proceedings will make no distinction between papers presented orally and those presented as posters. Would this paper make for a better oral or poster presentation?
  • Recommendation for Best Paper Award (NO or Yes): Do you think this paper should be considered for a Best Paper Award?

4. Justification for Best Paper Award Recommendations (non-mandatory):

To help the best paper committee make its selection, if you selected yes above, please write a sentence or two about what makes this paper particularly worthy. What is particularly good about this paper?

5. Confidential Comments (non-mandatory):

You may wish to withhold some comments from the authors and include them solely for the committee’s internal use. For example, you may want to express a very strong (negative) opinion on the paper, which might offend the authors in some way. Perhaps you wish to write something which would expose your identity to the authors. Or, you may have ethical concerns that Area Chairs/PC Chairs should be aware of?