When using LLM-as-a-judge, practitioners often use greedy decoding to get the most likely judgment. But we found that deriving a score from the judgment distribution (like taking the mean) works better! ❌LLM-as-a-judge with greedy decoding 😎Using the distribution of the judge’s labels
Victor Wang@victorwang37.bsky.social · last yr.LLM judges have become ubiquitous, but valuable signal is often ignored at inference. We analyze design decisions for leveraging judgment distributions from LLM-as-a-judge: 🧵 (w/ Michael J.Q. Zhang, @eunsol.bsky.social)
