Qwen3-1.7B was trained with a judge that calibrates its own confidence instead of an answer key. It reached 76.8% on the MATH500 benchmark, matching the 76.9% achieved with ground-truth answers S¹. The result, posted to arXiv on 20 November, challenges the assumption that reinforcement learning needs correct answers to grade against.

My read: The number I would watch is the gap on hard problems, where the method lifted accuracy from 44.8% to 51.7% S¹. That is where reference-free training should help most, and the paper reports it does. I am skeptical of the "no sign of reward hacking" claim because it is an absence reported by the same team that built the system, and the paper has not been peer-reviewed. The calibration error of 0.074, half the 0.141 of a standard softmax judge on the same 27B backbone, is the more convincing detail S¹.

The problem with trusting a judge

Reinforcement learning teaches a model by rewarding good outputs and penalising bad ones. For maths, the reward is simple: check the answer. For summarising a legal document, debugging code, or drafting a policy memo, nobody has an answer key. The standard workaround is to ask a larger language model to judge the output, but these judges rarely know how confident they should be. A judge that says "this is correct" with 99% certainty when it is actually right only 70% of the time will teach the model to produce outputs that look good to the judge rather than outputs that are actually good. Researchers call this reward hacking, and it is one of the main reasons RLHF, the technique behind ChatGPT's polish, is expensive and hard to control.

A judge that knows when it is wrong

The paper, listed as arXiv:2611.25135, describes a method called RLCDR, short for Reinforcement Learning from Calibrated Decision Rewards S¹. The authors took a language model and converted it into a decision model that answers typed questions about a response with a probability distribution, all in a single forward pass. They debiased it over option orders, the tendency of judges to prefer the first or last choice regardless of content, and calibrated it with about a hundred labels. The result is a judge that reports how sure it is, and that self-reported confidence tracks its actual accuracy.

Code for the method is on GitHub under an MIT licence, in a repository called eve-rlcd with 21 stars and two contributors, the most active being anthony-at-pieces with 65 contributions P³.

Training Qwen3-1.7B with GRPO, a reinforcement learning algorithm, on the MATH dataset using this reference-free reward raised accuracy on the MATH500 test set from 72.2% to 76.8%, a gain of 4.65 points with a p-value below 0.0001 S¹. That score matches training on the answer key, which reached 76.9%. A random reward signal gained only 0.65 points. That confirms the improvement comes from the judge's actual signal rather than the act of training itself S¹.

MATH500 accuracy by reward type Qwen3-1.7B

The gain grew with difficulty. On the hardest problems in MATH500, accuracy rose from 44.8% to 51.7% S¹. The judge stayed calibrated on the policy's own changing outputs, with an expected calibration error of 0.074, half the 0.141 of a standard softmax judge running on the same 27-billion-parameter backbone S¹.

Without any step-level training, the same model also located the first wrong step in a solution better than a dedicated 7-billion-parameter process reward model, scoring 0.772 versus 0.722 on ProcessBench F1 across 20 out of 20 splits S¹.

Where it breaks, and who tries it next

The method has a clear limit. When the authors trained only on problems the base model mostly fails, the reference-free reward still improved the policy by 1.9 MATH500 points (p = 0.03), but it trailed the answer key S¹. The authors attribute this to the judge's probabilities becoming overconfident on that distribution, a known failure mode for calibrated models pushed outside their training data.

The paper is not peer-reviewed. Every result is self-reported, and all experiments use a single model size, Qwen3-1.7B, on a single dataset, MATH. Whether the method generalises to larger models or other domains remains untested. The authors state six hypotheses in advance and test each with a control designed to falsify it, a level of rigour unusual for a preprint S¹.

Teams running reinforcement learning on tasks without answer keys, such as code review, document summarisation, or multi-turn agent evaluation, are the natural first users. A team that already has a judge model in their RLHF pipeline could swap in a calibrated decision model and check whether reward hacking drops. The code repository has been public since September 2026 P³. For teams already combining training and RL tooling, the method slots into existing pipelines; the torchtitan framework unified AI training and RL on PyTorch 2.15 on the same day this paper appeared, and Flide showed decision models running faster than vLLM earlier in November, both of which lower the cost of adding a calibrated judge.

The next checkpoint is whether independent labs reproduce the 76.8% figure on Qwen3-1.7B, or test the method on a larger model where calibration may behave differently.


Sources: S1 — Calibrated Decision Rewards: Training Language Models with a Judge Tha · P2 — 1The policy improves because of the reward. Qwen3-1.7B trained with GR · P3 — anthony-maio/eve-rlcd · P4 — 1The policy improves because of the reward. Qwen3-1.7B trained with GR · P5 — HuggingFaceH4/MATH-500 · Datasets at Hugging Face


Written from 5 sourced items, 4 of them primary.