GRPO

GRPO — Group Relative Policy Optimization — is a reinforcement-learning algorithm that scores each attempt against the other attempts in its own group rather than against a learned value function: the model generates several independent tries at the same problem, and an attempt’s advantage is how it did relative to its siblings. That removes the critic network — and with it a whole model to train — at the price of needing the group to disagree: a problem the policy always solves, or never solves, yields identical rewards across the group and no gradient at all.

---

The no-disagreement clause is not a footnote; it is the algorithm’s operating constraint, and it shapes data curation upstream. AfterQuery’s Terminal-Bench pipeline selects RL tasks where the SFT checkpoint solves 10–80% of attempts precisely because “these are the tasks where GRPO can compute meaningful advantages: some attempts succeed, some fail, and the model can learn from the difference” — and their per-test fractional reward exists for the same reason, to manufacture disagreement where binary grading would report uniform failure. Group size there: 16 attempts per task per step.