Updates · one entry per run

What changed, and what the numbers said.

Every run is pre-registered in PROTOCOL.md before a model is called and published whichever way it falls. This page is the short version, newest first; the long version is each run's README. Atom feed.

Run 6B · positive

When the gate says "this PR duplicates that one", it is right 98 times in 100. Half the time the right move is a link, not a close.

Run 3 found that most agent-PR rejections are coordination — someone else already landed it — and runs 4–5 built a PR memory that holds a PR when a cheap model says it is the same change as an open one. Nobody had checked whether those holds were right. Run 6B did: 100 PRs that the 4B model holds at duplicate_of ≥ 0.7 with a 20-candidate pool, stratified by what later happened to them, labelled blind by the author over five days (A, its top candidate B, the repository; no score, rank or model name) with three labels: duplicate (B makes A unnecessary), pointer (A's author should have looked at B first — backport, edge→stable promote, competing fix, follow-up, same area), unrelated. Result: 98 of 100 holds were right (bar 0.60) and 44 were the same change (bar 0.30). The 54 pointers are the finding: 28 are the same change moving to another branch, 8 are two independent fixes of one bug, 6 are follow-ups or repro↔fix pairs. When the held PR was later merged, six in seven holds are pointers; when it was closed unlabelled, three in four are the same change. A second blind labeller agrees at κ 0.77 and disagrees only on where duplicate ends and pointer begins; the 12 disagreements are adjudicated in the open. The pool size turned out not to be the lever (the verdict-level bar fails at every k; k = 10 is the default) and the whole thing costs $0.0004 per PR. Consequences: the dedup rule stays a hold; a hold never closes a PR — even at a 0.9 threshold two in five holds are pointers; and the next question the model has to answer is the relation between two PRs, not just whether they are the same. That is run 7.

holds that were right · bar 0.600.98
same change · bar 0.300.44
pointers, not duplicates54
second labeller, κ0.77
cost per PR, k=20$0.0004
results/run6 part Btablesadjudicated labels
Reading · context

×1 is not a projection any more. It is September.

The Claude Code team described their review stack (Osmani, Lunt, Wang, 15 Oct). Three things in it matter for this benchmark. The volume: daily landed changes in their monorepo went from about 2,500 in May to over 30,000 in September — the base rate every load test here calls ×1, measured rather than assumed, with merged PRs doubling while the revert share fell 40%. The shape: a classifier sets each change's risk tier from the change itself, a second classifier asks whether a human is needed anyway, owner-listed sensitive paths and CI workflow files always get a person, agent approvals never count as human ones, and an approval covers exactly one commit. That is a typed, calibrated decision in front of trunk with a policy on top — the design the six runs above test, independently arrived at. The gap: the post treats duplicates only as "don't repost a finding" and review loops as a convergence problem (one PR went 50+ rounds before a three-round cap; mean rounds to pass 2.08 → 1.32). Two agents opening the same change twice — the coordination failure behind 67% of the closures in run 3 — is not in it. Run 6B measures exactly that: of 100 PRs the gate holds as duplicates, how many a blind human agrees are. Two changes follow from the post: the gateway gets a max_rounds cap, and any prompt change in run 7 has to hold its score on the revert set before it is kept.

Seven review layers from the post; layer five highlighted as the decision mergeone measures
landed changes / day, Sept30,000+
humans per PR, target≤ 1
rounds to pass, after cap1.32
run 3 closures that are coordination67%
the postROADMAP
Run 6A · negative

The open weights are not the limit. One GPU is.

Run 5 ended with Perplexity's hosted API refusing 71% of calls above 10 requests/s per organisation. The weights are Apache-2.0, so run 6A served the same 4B decider ourselves — vendor inference code, no quantisation, one rented NVIDIA L4 at $0.59/h, benchmark client on the same box. It answers like the hosted endpoint on 95.2% of argmaxes but not within the 0.02 mean |Δp| bar (0.047), so by the protocol it is a different model; quality is a confidence interval away (AUC 0.747 vs 0.758; pair task 0.840 vs 0.842); the model card's one-question-per-request advice gains +0.000 AUC for ten times the calls; and the server saturates at about 2.2 requests/s — six times the 30k-decisions/day base rate, an order of magnitude under the 100× bar. Self-hosting removes the vendor's rate limit and replaces it with capacity planning. A second pass showed the lever: length-aware batching changed nothing on either card (a request takes as long inside a batch as alone), but one H100 carries 30× the base rate at 160 ms p99 with zero errors — faster than the hosted endpoint — and still times out 30% of calls at 100×. $0.106 per thousand PRs at that rate, 4.6× the hosted list price. Cost of all three measurements: about $6.

argmax agreement95.2%
mean |Δp| · bar 0.020.047
AUC vs hosted0.747 / 0.758
L4 saturates at×10
H100 at ×30, p99160 ms
$/1k PRs · L4 ×1 / H100 ×300.47 / 0.11
p99 latency per decision versus load for six decision-model rows; each line runs as far as every call was answered
results/run6leaderboard
Run 5 · mixed

A flag means more review. The stack rule chose wrong.

Retrieval v2 (shared issue references, directory overlap, weights fit on validation only) lifts recall@5 from 0.55 to 0.66 against a 0.70 bar, and the Go store answers in 3.3 ms p99 against 20. Run 3's exploratory finding became a result: calibrated needs_human_review predicts which agent PRs go through two or more force-push rounds at AUC 0.76 for Jev and for the 4B open-weight pplx-flash (bar 0.70). The ten-question stack missed its 0.75 bar for every System One model because the pruning rule, chosen on validation as pre-registered, dropped hard_to_revert — the strongest single question on test (0.85). Declared assumptions could not be judged: maintainers wrote a "depends on / conflicts with" comment on 9 of 2,196 PRs. Both hosted Perplexity models held p99 flat from 1× to 100× and then lost 71% of calls to HTTP 429.

recall@5 · bar 0.700.66
flag ⇒ iterated AUC · bar 0.700.76
store p99 · bar 20 ms3.3 ms
hosted errors at 100×71%
results/run5
Run 4 · mixed

A cheap model can tell two PRs are the same change. Finding the pair is the bottleneck.

Asked whether PR A and PR B are the same change, three System One models clear the 0.80 bar on held-out repositories — Perplexity's open-weight decider 0.87, its 4B flash 0.84, Jev 0.83 — at under half a second and under $0.0005 per PR; the LLM judge (0.76) does not. The retrieval that proposes the pairs finds only 55% of maintainer-confirmed duplicates in its top 5 (bar 60%), so at the verdict level the dedup rule moves the pass group's bad-outcome rate from 0.41 to 0.36 against a 0.23 bar. Four of five hypotheses fail; PR memory ships opt-in.

same_change AUC · bar 0.800.83–0.87
recall@5 · bar 0.600.55
hypotheses passed1 / 5
results/run4
Run 3 · negative

The gate cannot see that someone else already landed the fix.

2,196 PRs opened by coding agents across 26 public repositories, held-out test of 705 PRs in 8 repositories. Does calibrated needs_human_review predict that a PR is closed unmerged or gets changes requested? ROC-AUC 0.55 for Jev, 0.55 for Haiku, 0.53 for rules, against a 0.70 bar. The labels explain it: of 200 closures classified by hand, 67% were superseded, duplicate or direction, 19% quality. Two things held — Jev at 100× agent-scale load (p99 417 ms, zero errors) and, exploratory only, the questions tracking review iteration at AUC 0.70–0.81. That second one became run 5's hypothesis; the first finding became run 4.

AUC · bar 0.700.55
closures that were coordination67%
Jev p99 at 100×417 ms
results/run3
Run 2 · positive

Five typed questions, one call, 347 ms.

Behaviour change, weakened tests, concurrency risk, message/diff match, needs-human: five noul questions per PR, scored against a strong judge. Jev meets the bar on all five (ROC-AUC 0.87–0.96) at 347 ms and $0.074 per thousand PRs; Haiku on four at 1,067 ms. Calibrated ECE under 0.05 for both. This is the run the gateway's default question set comes from.

AUC vs judge0.87–0.96
Jev p50347 ms
$/1k PRs0.074
results/run2
Run 1 · negative

Nothing in a diff predicts a 30-day revert.

40,000 Kubernetes merges, positives are PRs reverted within 30 days (base rate 0.75%). Rules, a gradient-boosted classifier, Haiku and Jev all land at PR-AUC 0.010 — chance. The signal is not in the pre-merge state, and the project stopped trying to find it there. What did hold: the load bar (100× base rate, zero errors) for the System One model, and calibration that stays honest when the answer is "no".

PR-AUC · base 0.00750.010
merges40,000
results/run1
Protocol frozen

gatebench: hypotheses, thresholds and data rules committed before any model call.

Candidates are System One decision models (Jev; from run 4, open-weight deciders), one small LLM as the judge baseline, and a rules floor. Frontier models are never candidates. Every deviation after the freeze is dated in CHANGELOG.md with a reason; thresholds are never edited after results exist.

PROTOCOL.md