Implementation judge: score self-directed diffs, auto-merge only above the bar #1772

Closed
opened 2026-11-02 14:32:19 +00:00 by sam · 1 comment
Owner

Stage 3 of the self-development loop. Wanted: extend the PR-review benchmark's judge model from reviewing PRs to scoring implementation diffs, calibrated against the maintainer's own merge decisions on recent PRs, so "better" is a number rather than a vibe. Auto-merge applies only when the gate is green, the adversarial review stage is clean, and the judge scores above an acceptance bar set during calibration (same pattern as the review bench's recall bar).

Done when: on a calibration set of recently merged and rejected PRs the judge's accept/reject agrees with the maintainer's decision at or above the agreed bar, and auto-merge is enabled for that qualifying class only.

Stage 3 of the self-development loop. Wanted: extend the PR-review benchmark's judge model from reviewing PRs to scoring implementation diffs, calibrated against the maintainer's own merge decisions on recent PRs, so "better" is a number rather than a vibe. Auto-merge applies only when the gate is green, the adversarial review stage is clean, and the judge scores above an acceptance bar set during calibration (same pattern as the review bench's recall bar). Done when: on a calibration set of recently merged and rejected PRs the judge's accept/reject agrees with the maintainer's decision at or above the agreed bar, and auto-merge is enabled for that qualifying class only.
sam added the mediumfeature labels 2026-11-12 15:51:28 +00:00
sam added a new dependency 2026-11-13 03:03:15 +00:00
Author
Owner

Closed: goes against the product plan in epic #1977. A loop where Archie develops its own codebase through PRs frames Archie as a coding agent; coding is a side effect, not the goal. Learning from outcomes is covered by proposals (#1995).

Closed: goes against the product plan in epic #1977. A loop where Archie develops its own codebase through PRs frames Archie as a coding agent; coding is a side effect, not the goal. Learning from outcomes is covered by proposals (#1995).
sam closed this issue 2026-11-14 14:45:11 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sam/archie#1772