Evaluating the Evaluators: Investigating LLM Judges for Personalized Writing Style Assessment
Reported by Apple Machine Learning Research1 source ↓
The paper was accepted at the Can We Trust AI Evaluation? Workshop at NeurIPS 2026. It reports that LLM judges’ agreement with human preferences varies by writing task and evaluation criteria.
Key points
- The study compared LLM judges with human preferences across four text generators and three long-form writing tasks
- Judge quality had weak correlation with general LLM ability measured by MMLU (Pearson’s ρ ≤ 0.2)
- LLM judges were more consistent and performed better with paired comparisons than independent scoring
- The strongest models did not most reliably reproduce human evaluations
Details
The researchers generated writing samples with four LLMs across three long-form tasks, then collected human labels and LLM judgments comparing pairs of responses. They assessed both general helpfulness and adherence to a personalized writing style.
The study found that judge performance varied with the task and evaluation criterion, and correlated weakly with MMLU results. The authors suggest that stronger models may react more to subtle details that human annotators and weaker models overlook.
Written from coverage as of
Images
- Apple Machine Learning Research