Note that this is a guest post written by Jenny Wang and Audrey You of the University of Auckland; they are the lead authors on the research below conducted by the University of Auckland and Multitudes.
We recently had the opportunity to present our paper, “Assessing Harmful Comments and Specificity in Code Review Feedback at Scale using Large Language Models”, at the ACM International Conference on the Foundations of Software Engineering (FSE 2026) in Montréal, Canada.
The paper was a collaborative project between the University of Auckland and Multitudes, and explored whether large language models (LLMs) can help us understand the quality of feedback developers give each other during code reviews.
View a one-pager summarizing the research here: bit.ly/FSE-one-pager
See a copy of the slides we shared at FSE here: docsend.com/view/gdy2snbieedbxu3w
View the full paper here: doi.org/10.1145/3803437.3805264
Code reviews are an important part of modern software development. They provide opportunities for developers to improve code, share knowledge and learn from one another. When feedback is clear and actionable, it can strengthen collaboration and team trust.
But feedback can also go wrong. Vague or unhelpful comments can make it difficult for developers to know what to change, while harmful or dismissive language can negatively affect collaboration and inclusion.
The challenge is scale. Manually analyzing hundreds of thousands of review comments isn't practical. This led us to ask: Can LLMs automatically assess the quality of code review feedback at scale?
We designed our study around two research questions:
We collected three datasets for different purposes: 573 potentially harmful open-source feedback threads, a general open-source dataset containing over 204,000 threads from 30 popular repositories, and an industry dataset containing over 105,000 threads from 45 clientele of Multitudes.
We manually labelled subsets of the data for sentiment and specificity. Sentiment was categorised as positive, neutral, or negative/harmful. Specificity ranged from highly specific feedback, which provides concrete guidance and reasoning, to highly unspecific feedback that provides no concrete suggestion for improvement. We also identified minimal reviews (most often a “looks good to me” or “LGTM”); we set the threshold for these as reviews containing fewer than 25 characters.
We then evaluated 11 LLMs using zero-shot prompting, meaning the models were not fine-tuned for our task. We used symbol tuning, replacing familiar labels such as “positive” and “negative” with arbitrary symbols, and used BAML to enforce structured outputs and identify invalid responses.
For sentiment classification, Claude Sonnet 3.5 performed best overall, achieving the best F1 score on the harmful feedback classification at 0.767 (see chart above) and a good F1 score on the general open-source dataset at 0.835.
However, the results also highlighted an important challenge: Models sometimes over-flagged firm but professional feedback as harmful. We saw three recurring types of errors: excessive nitpicking across a thread, harmful comments buried deep within otherwise neutral discussions, and neutral but firm feedback being interpreted as negative.
Specificity was a more difficult classification task. Mixtral 8x7B achieved the best F1 score of 0.674 (see chart above), with most errors occurring around the boundary between neutral and highly specific or highly unspecific feedback. This isn't entirely surprising: specificity can be subjective, and the point at which a comment becomes sufficiently concrete can be difficult for even human annotators to agree on.
One particularly interesting example was a comment saying, “you spelt maintainers wrong in title.” Six of the 11 models classified this as highly specific, despite the human label being highly unspecific. The comment identifies a concrete issue, but without access to the pull request's metadata or surrounding context, the models could not recognize that the issue was relatively trivial.
We then applied our best-performing model to the larger datasets.
The results showed distinct patterns (see chart above). Open-source feedback was predominantly neutral specificity, accounting for around 47% of feedback. Industry feedback was more polarized: 29% was highly specific, but 22% was minimal, with many short approvals such as “LGTM” or “Looks good.”
Harmful feedback was relatively rare in both settings, but it was 3x higher in open source repositories (1.9%) than in industry repositories (at 0.6%). It’s worth paying attention to this despite the low frequency because previous research suggests that infrequent harmful interactions can still have lasting effects.
One of the most valuable parts of the research was seeing what happened when these classifications moved beyond a research dataset and into a real product.
The feature was deployed in Multitudes's developer productivity platform and remained one of its most-used features for months. You can see the first iteration of the feature below.
Specificity produced particularly useful insights. In one organization, the analysis revealed that a senior developer was consistently giving much more specific feedback than they received in return (this came out in Multitudes's 1:1 insights page). This highlighted a potential mentorship gap and prompted discussions about how the team could provide more feedback to that developer.
Sentiment was more complicated.
Users were uncomfortable seeing their own comments labelled “negative”, even when the model had high confidence. This reinforced something we had already seen in our evaluation: LLM classifications are not always definitive, and for sensitive topics, the accuracy bar needs to be very high. In response, the Multitudes team reframed “negative” as “needs attention”, changed the visual indicator from red to yellow, and allowed users to relabel comments when they disagreed. They also added an insight at the top of the chart to help interpret the results, and provided industry benchmarks for highly specific feedback and minimal reviews, so that people wouldn't think that the goal is all specific feedback or zero minimal reviews. See below for what the second iteration of the feature looked like.
The experience taught us an important product lesson: Uncertainty needs to be communicated clearly when AI is part of the evaluation.
The discussion after our presentation was one of our favorite parts of the conference.
One person asked how we designed the specificity prompt, given that specificity is inherently subjective. Our response was that specificity was defined using a clear rubric: A comment is labeled highly specific if it includes resources like links (to learn more) or code snippets with associated reasoning, highly unspecific if it contains no suggestion for improvement at all, and neutral specificity if it sits in the middle: Containing a recommended change but lacking reasoning.
Someone else asked whether there might be a relationship between negative sentiment and specificity. Our study did analyse these dimensions together: we used a hierarchical set of feedback categories that allowed comments to be classified as harmful, highly specific, neutral, highly unspecific, or minimal. However, harmful feedback was given priority in this categorisation, so the published results do not directly show how much overlap there was between harmful feedback and each level of specificity. This leaves an interesting direction for future work: examining the combinations more directly. For example, are harmful comments more likely to be highly unspecific, or can harmful feedback also be highly specific and technically actionable? It would also be interesting to investigate whether these different combinations have different effects on outcomes such as whether developers act on the feedback, how long review discussions take, or how recipients perceive the feedback.
Another idea was to incorporate social interactions outside the review thread itself to improve classification. While richer context would likely help models interpret intent more accurately, we also noted that there are inherent limits: LLMs will never have access to the full social and organizational context behind every comment. This makes it important to design systems that are aware of these constraints and to interpret model outputs with appropriate uncertainty.
These questions reinforced that there is still a lot to explore.
There are several directions we'd like to investigate. Few-shot prompting and calibration could improve performance on borderline specificity cases. Incorporating pull request context, including code diffs and metadata, could help models interpret comments more accurately.
We also have finer-grained harmful feedback categories such as judgmental, vague and harsh language, that we have labelled and predicted but not yet evaluated systematically (these categories came from our literature review on feedback quality; citations in the “for more” section below).
Finally, repeating the study with newer LLMs could reveal how much classification performance has improved as models evolve.
We'd also like to thank Multitudes, and our co-authors Youxiang Lei, Lauren Peate, and Kelly Blincoe for their support throughout the project.
Get useful insights on team performance and culture, along with updates on our product!
© Copyright 2026 Multitudes Limited. All rights reserved.
.svg)
.svg)
.svg)
.png)
