Implementation judge: score self-directed diffs, auto-merge only above the bar #1772
Notifications
Due Date
No due date set.
Blocks
Reference: sam/archie#1772
Reference in new issue
Block a user
Stage 3 of the self-development loop. Wanted: extend the PR-review benchmark's judge model from reviewing PRs to scoring implementation diffs, calibrated against the maintainer's own merge decisions on recent PRs, so "better" is a number rather than a vibe. Auto-merge applies only when the gate is green, the adversarial review stage is clean, and the judge scores above an acceptance bar set during calibration (same pattern as the review bench's recall bar).
Done when: on a calibration set of recently merged and rejected PRs the judge's accept/reject agrees with the maintainer's decision at or above the agreed bar, and auto-merge is enabled for that qualifying class only.
Closed: goes against the product plan in epic #1977. A loop where Archie develops its own codebase through PRs frames Archie as a coding agent; coding is a side effect, not the goal. Learning from outcomes is covered by proposals (#1995).