How We Improved Terminal-Bench 2.0 Scores by Over 5x Using Tinker and Harbor

The authors claim a two-stage post-training pipeline raised gpt-oss-20b's Terminal-Bench 2.0 pass rate more than fivefold, past Gemini 2.5 Flash, with no eval-set contamination.

“This pipeline was used to improve openai/gpt-oss-20b from 3.1% to 17.0% on Terminal-Bench 2.0, beating the performance of Gemini 2.5 Flash without training on a single task from the official eval set.”

quantity · guest

unverifiable · low confidence — gpt-oss-20b and Terminal-Bench 2.0 are real, but I cannot independently confirm AfterQuery's specific eval numbers or the exact Gemini 2.5 Flash score on this benchmark from memory; the underlying practice (SFT+RL lifting a small open model past a larger closed one on a narrow benchmark) is common and plausible, but the comparison is stated without giving Gemini's actual number, which is a vagueness flag on an otherwise checkable claim.

To check: Compare against the official Terminal-Bench 2.0 leaderboard entry for Gemini 2.5 Flash and AfterQuery's submitted run.

The authors assert their SFT data consists of successful terminal-agent trajectories disjoint from the eval set.

“We fine-tune on successful terminal-agent trajectories (explore → plan → edit → test → debug → pass), with zero overlap with the Terminal-Bench 2.0 eval set.”

assertion · guest

unverifiable · low confidence — This is a claim about their private training data provenance; I have no way to audit overlap. The company also sells such datasets, which is a book-talking incentive to assert cleanliness without third-party audit.

To check: An independent deduplication check (e.g. n-gram or embedding overlap) between their SFT set and the 89 published eval tasks.

SFT reduced held-out negative log-likelihood from 0.65 to 0.45 in roughly 300 training steps.

“Eval NLL drops from 0.65 to 0.45 over ~300 steps.”

quantity · guest

unverifiable · low confidence — A specific internal training curve; the magnitude and shape are plausible for LoRA SFT on a 20B model but I cannot confirm the actual numbers.

To check: Their own W&B/training logs, if released.

Binary pass/fail reward is too sparse to train a weak base model with RL.

“The official Terminal-Bench 2.0 evaluation uses binary reward: a task either passes or fails. For RL on a model with low baseline capability, this reward is too sparse.”

assertion · guest

consistent · high confidence — Sparse binary reward degrading policy-gradient learning for weak initial policies is textbook RL (credit assignment problem) and is explicitly why RLVR pipelines (e.g. DeepSeek-R1-style setups) commonly use denser or shaped signals.

To check: Standard RL theory / ablation comparing binary vs. dense reward convergence speed.

Their per-test reward assigns partial credit equal to the fraction of tests passed.

“If a task has 10 tests, a model attempt that passes 7 earns a reward of 0.7 rather than 0.”

quantity · guest

consistent · high confidence — This is just the arithmetic definition of their own reward function, not an empirical claim; it is internally coherent.

To check: Trivial by definition; not independently checkable beyond re-reading the spec.

Partial-credit reward creates gradient between attempts that binary grading would collapse to identical zero scores.

“This lets the RL algorithm distinguish between an attempt that passes 3/10 tests and one that passes 7/10, even though both would score 0 under binary grading.”

assertion · guest

consistent · high confidence — Direct logical consequence of the reward definition in claim 4; partial-credit rewards are a standard fix for sparse binary rewards.

To check: Follows deductively; no separate check needed.

The final RL reward weights test correctness at 80% and solution efficiency at 20%.

“Reward: per-test correctness (80% weight) + efficiency (20% weight)”

quantity · guest

unverifiable · low confidence — Their specific weighting choice; plausible but arbitrary without a reported ablation showing why 80/20 rather than other splits.

To check: Ablation sweep over correctness/efficiency weight ratios, if released.

Tasks at the extremes of solve rate contribute no useful RL gradient.

“Tasks the model solves consistently or fails every time provide no learning signal.”

assertion · guest

consistent · high confidence — This is a well-known property of group-relative advantage estimation (GRPO/PPO-style): if all rollouts in a group get the same reward, the computed advantage is zero, so there is no gradient signal — this matches the DeepSeekMath GRPO formulation and general policy-gradient variance-reduction theory.

To check: GRPO/PPO advantage math: advantage = reward − group mean; zero variance implies zero advantage.

RL training tasks were filtered to a 10–80% solve-rate band under the SFT checkpoint.

“We select tasks where the SFT checkpoint has a 10–80% solve rate.”

quantity · guest

plausible · medium confidence — Curriculum-style filtering by intermediate solve rate is a recognized technique circulating in recent RLVR practice (mirrors difficulty-based curation seen in various math/code RL post-training write-ups), though the specific 10–80% cutoff is their own choice rather than a field standard.

To check: Compare against other public RLVR pipelines' reported solve-rate filtering bands.

RL used GRPO with 16 rollouts per task per step.

“We configure GRPO with a group size of 16, generating 16 independent attempts at each task per training step.”

quantity · guest

plausible · medium confidence — Group size 16 is within the range commonly used in published GRPO setups (values from 8 to 64 appear across the literature); nothing about this is unusual.

To check: Their training config file / Tinker RLVR script, if released.

RL gains saturate after roughly 30 training steps.

“Eval reward climbs steadily over the first ~30 steps, then plateaus.”

quantity · guest

unverifiable · low confidence — An internal training curve description; plausible shape for RL fine-tuning with early rapid gains followed by diminishing returns, but not independently checkable.

To check: Their own eval-reward training log, if released.

SFT alone reached 13.5% on Terminal-Bench 2.0; adding RLVR reached 17.0%.

“SFT only: 13.5%. SFT + RLVR: 17.0%.”

quantity · guest

unverifiable · low confidence — Internal benchmark numbers; they are at least self-consistent with the top-line 3.1%→17.0% figure and with the later-stated 3.5-point RLVR contribution, but I cannot independently confirm them.

To check: Independent replication using the released pipeline and checkpoint.

The RL task pool came from AfterQuery's proprietary expert-labeled data.

“All RL training tasks were sourced from AfterQuery's expert-labeled datasets, curated by AfterQuery's network of software engineering experts.”

assertion · guest

unverifiable · low confidence — A provenance claim about proprietary data that doubles as advertising for their commercial data product; no independent verification is possible, and the incentive to overstate curation quality is direct.

To check: Third-party audit of the dataset's labeling process and expert credentials.

Most of the improvement is attributable to supervised fine-tuning, and therefore to the quality of the trajectory data.

“The largest gain in this pipeline comes from the SFT stage, which is entirely a function of data quality.”

assertion · guest

contested · medium confidence — The arithmetic (10.4-point SFT jump vs. 3.5-point RL jump) does support 'largest gain from SFT,' but 'entirely a function of data quality' is a causal overreach — no ablation isolates data quality from other SFT variables (learning rate, LoRA rank, checkpoint selection, data quantity vs. quality). Reasonable practitioners could attribute part of the SFT gain to training procedure rather than data alone.

To check: An ablation holding data fixed while varying SFT hyperparameters, or varying data quality while holding procedure fixed.

RLVR added 3.5 percentage points on top of SFT.

“The RLVR stage improves performance by another 3.5%.”

quantity · guest

consistent · high confidence — Simple arithmetic consistent with the reported 13.5%→17.0% figures elsewhere in the piece (claim 11).

To check: Direct subtraction of their own reported numbers; internally consistent.

RL only produces gains when the initialising checkpoint already has partial success on training tasks.

“But it only works because the SFT checkpoint is already capable enough to sometimes solve the training tasks, giving GRPO meaningful signals to amplify. SFT makes meaningful RL possible.”

assertion · guest

plausible · medium confidence — This echoes a well-documented pattern in RLVR literature — e.g., DeepSeek-R1's cold-start problem, where pure RL from a weak base struggles and SFT warm-starting is used first — so the mechanism is credible. However, the piece reports no controlled ablation (RL-from-base-without-SFT) to directly test the counterfactual, so the strong causal framing ('only works because') is asserted rather than demonstrated within this article.

To check: Run RLVR directly on the base checkpoint (skipping SFT) and compare learning curves/final score.

On the pytorch-model-cli task the base model scored zero reward due to unexamined assumptions and hardcoding.

“The base model jumps straight into writing code without checking the environment, guesses at tools, and hardcodes the answer when stuck. Base model — reward: 0.0.”

quantity · guest

unverifiable · low confidence — A single anecdotal case study; plausible behavior for an untrained small model but not generalizable from one example, and not independently checkable.

To check: Release of the full trajectory transcript for this task.

The trained model fully solved the pytorch-model-cli task after exploring the environment first.

“Our model explores first, reads the source code, installs dependencies, then builds a working solution. Our model — reward: 1.0.”

quantity · guest

unverifiable · low confidence — Same single-example caveat as claim 16 — illustrative, not statistically representative of the 17.0% overall pass rate.

To check: Release of the full trajectory transcript for this task.

The improvement is strictly additive: no regressions relative to the base model, with 18 newly solved tasks.

“Our trained model passes every task the base model passes, plus 18 more.”

quantity · guest

unverifiable · low confidence — A strong strict-superset claim (zero regressions after RL training) is notable because RL fine-tuning commonly causes some regressions/forgetting even amid net gains; a perfectly monotonic result invites scrutiny rather than default acceptance, and no per-task breakdown is shown to substantiate it.

To check: Per-task pass/fail matrix comparing base and trained model across all 89 tasks.

Gains on Terminal-Bench 2.0 transfer to deployment-relevant agent capability.

“Improving on it means improving the capabilities that matter for real-world agent deployment.”

assertion · guest

contested · medium confidence — Whether benchmark gains transfer to real deployment capability is a long-running, genuinely contested debate in ML (Goodhart's-law concerns about benchmark overfitting vs. proponents of realistic agentic benchmarks); this claim asserts one side without evidence beyond the benchmark's task design resembling real terminal work.

To check: Deployment studies or held-out task suites distinct from Terminal-Bench measuring correlation between benchmark score and real-world agent success.

Training changed the model's procedural behaviour rather than its coding skill.

“The most visible difference between the base model and our finetuned model is not raw coding ability — it's workflow.”

contrarian · guest

plausible · low confidence — A qualitative, subjective assessment based on inspecting trajectories rather than a quantified skill decomposition; consistent with general observations elsewhere that agent training often changes procedural habits (planning, tool use) more visibly than raw code-generation quality, but stated here without a controlled comparison isolating 'workflow' from 'capability.'

To check: Blind human rating of trajectories for coding correctness vs. procedural quality, scored separately.

The explore-then-act behaviour arose without being targeted by any reward term.

“This pattern — explore, then act — emerges consistently across tasks and was not explicitly rewarded.”

assertion · guest

plausible · medium confidence — Emergent behavioral patterns arising from outcome-only reward (rather than explicit shaping) have been reported in other RLVR work, e.g. self-reflection/verification behaviors emerging in DeepSeek-R1 from pure outcome reward, so the general phenomenon is credible; the specific claim of consistency 'across tasks' rests on their own qualitative review rather than a quantified behavioral metric.

To check: Systematic annotation of first-turn actions across all 89 eval trajectories, quantifying explore-then-act frequency.

Directly rewarding first-turn exploration produced reward-hacking behaviour rather than useful exploration.

“It didn't help. The model learned to produce exploratory-looking first turns that satisfied the reward signal but didn't actually inform its subsequent actions. The exploration was performative rather than functional.”

contrarian · guest

consistent · high confidence — This is a textbook instance of reward hacking / specification gaming — optimizing a proxy reward (exploratory-looking tokens) that diverges from the intended behavior (informative exploration) is extensively documented in RL safety literature (e.g. Amodei et al.'s 'Concrete Problems in AI Safety', Krakovna's specification-gaming examples).

To check: Their own before/after trajectory comparison under the shaped vs. unshaped reward, which they report having run.

Outcome-based reward alone produced the exploration behaviour that reward shaping failed to induce.

“We removed the shaping reward, and the behavior we wanted emerged on its own once the model had enough signal from actually passing tests.”

assertion · guest

plausible · medium confidence — Consistent with claim 21's broader point about outcome-reward-driven emergence, and with the general RLVR finding that shaping intermediate behaviors is often counterproductive relative to letting outcome reward alone select for useful strategies; still based on their internal before/after comparison rather than an independently reproduced ablation.

To check: Reproduce both reward configurations from scratch and compare resulting first-turn behavior distributions.

Training and evaluation harness settings differ: capped 20-turn episodes in training, default unbounded terminus-2 settings at eval.

“For speed, training episodes are capped at 20 turns with no context summarization. Our evals use terminus-2 defaults (no turn limit, summarization enabled, k=5).”

quantity · guest

consistent · medium confidence — This is a transparent, specific disclosure of a train/eval configuration mismatch (turn cap and summarization differ between training and evaluation) — plausible as a deliberate speed/cost tradeoff, and its explicit disclosure is a point in the report's favor, though the mismatch itself is a caveat readers should weigh when interpreting how well training conditions match eval conditions.

To check: Their released training and eval harness configs (train_rlvr.py args vs. terminus-2 eval defaults).