Benchmark acceptance run #131

Closed
opened 2026-10-27 13:35:05 +00:00 by sam · 3 comments
Owner

Golden micro-recall at least 0.70. See docs/prds/pr-review-agent.md.


Imported from beads archie-core-afbk.6 · GitHub #1199 · type task · priority P1 · created 2026-09-25

Golden micro-recall at least 0.70. See docs/prds/pr-review-agent.md. --- _Imported from beads `archie-core-afbk.6` · GitHub #1199 · type task · priority P1 · created 2026-09-25_
sam added the hightaskin-progress labels 2026-10-27 13:35:05 +00:00
sam self-assigned this 2026-10-27 13:35:05 +00:00
Author
Owner

sam wrote (originally 2026-10-02):

Triage 2026-10-02 @ 11974196: PARTIAL. Harness + metric + gate and external-PR sourcing are SHIPPED (1c7dd08f harness, bae53bcb live pipeline, dabe7c88 current form; prsource 7ddaed38, daemon prefetch 2142e446). The bead's own acceptance run has NOT happened: zero acceptance artifacts exist. REMAINING: run the 38-PR one-blind-deep acceptance (task bench:prreview) and record recall >= 0.70 durably -- bin/ is gitignored, so a summary.json there is not tree evidence. BLOCKER is model availability, not code: the PRD names GLM-5.2 for review+classification and anthropic/claude-sonnet-4.6 as judge, and substitutions do not satisfy the criterion (docs/guides/pr-review-benchmark.md:28). Needs GH_TOKEN with read access to the 38 upstream Martian PRs. This bead gates em7s.8.

_sam wrote (originally 2026-10-02):_ Triage 2026-10-02 @ 11974196: PARTIAL. Harness + metric + gate and external-PR sourcing are SHIPPED (1c7dd08f harness, bae53bcb live pipeline, dabe7c88 current form; prsource 7ddaed38, daemon prefetch 2142e446). The bead's own acceptance run has NOT happened: zero acceptance artifacts exist. REMAINING: run the 38-PR one-blind-deep acceptance (task bench:prreview) and record recall >= 0.70 durably -- bin/ is gitignored, so a summary.json there is not tree evidence. BLOCKER is model availability, not code: the PRD names GLM-5.2 for review+classification and anthropic/claude-sonnet-4.6 as judge, and substitutions do not satisfy the criterion (docs/guides/pr-review-benchmark.md:28). Needs GH_TOKEN with read access to the 38 upstream Martian PRs. This bead gates em7s.8.
Author
Owner

sam wrote (originally 2026-10-10):

Runner restored (commit 5f077d20 on branch worktree-agent-ab430d62419672912, unmerged). The old runner and task had been deleted as orphaned. To run the acceptance: set the default model alias in ~/.config/archie/config.toml (or pass flags), then 'task bench:prreview -- -model ALIAS_OR_PROVIDER/MODEL -judge-model ALIAS_OR_PROVIDER/MODEL'. No GH_TOKEN needed (public PRs; token only lifts API rate limit). Commit the docs/benchmarks/pr-review-*.md report it writes. See docs/guides/pr-review-benchmark.md. Acceptance run itself is not done.

_sam wrote (originally 2026-10-10):_ Runner restored (commit 5f077d20 on branch worktree-agent-ab430d62419672912, unmerged). The old runner and task had been deleted as orphaned. To run the acceptance: set the default model alias in ~/.config/archie/config.toml (or pass flags), then 'task bench:prreview -- -model ALIAS_OR_PROVIDER/MODEL -judge-model ALIAS_OR_PROVIDER/MODEL'. No GH_TOKEN needed (public PRs; token only lifts API rate limit). Commit the docs/benchmarks/pr-review-*.md report it writes. See docs/guides/pr-review-benchmark.md. Acceptance run itself is not done.
sam added a new dependency 2026-10-27 13:41:45 +00:00
sam added a new dependency 2026-10-27 13:41:45 +00:00
sam added a new dependency 2026-10-27 13:41:49 +00:00
Author
Owner

Triaged: no code change is needed for this. This is an acceptance run, not a build. The benchmark harness already exists and is complete: task bench:prreview runs archied prbench (internal/app/prbench), which reviews the 38 embedded Martian PRs in internal/app/prbench/martian.json with the production pipeline, judges posted comments against the goldens, and writes docs/benchmarks/pr-review-.md. The 0.70 golden micro-recall bar from the PRD is implemented as acceptanceRecall in internal/app/prbench/prbench.go, and a full run reports PASS/FAIL against it; docs/guides/pr-review-benchmark.md documents the command, flags and acceptance rule.

What remains is to execute that command with the operator's configured models (the reviewer and an independent judge) and commit the generated report; that needs the instance's config at ~/.config/archie/config.toml and outbound network, neither of which is a repository change. No code path is missing, stubbed or broken as far as reading shows.

If the full run reports micro-recall below 0.70 or any failed problem, that report is the evidence for a follow-up code change to the review pipeline; that would be a separate item rather than this acceptance run. Note the body's reference to docs/prds/pr-review-agent.md does not resolve in this tree (PRDs for landed work are absent), but the acceptance criterion it stands for is the one implemented and documented above.

Triaged: no code change is needed for this. This is an acceptance run, not a build. The benchmark harness already exists and is complete: `task bench:prreview` runs `archied prbench` (internal/app/prbench), which reviews the 38 embedded Martian PRs in internal/app/prbench/martian.json with the production pipeline, judges posted comments against the goldens, and writes docs/benchmarks/pr-review-<timestamp>.md. The 0.70 golden micro-recall bar from the PRD is implemented as `acceptanceRecall` in internal/app/prbench/prbench.go, and a full run reports PASS/FAIL against it; docs/guides/pr-review-benchmark.md documents the command, flags and acceptance rule. What remains is to execute that command with the operator's configured models (the reviewer and an independent judge) and commit the generated report; that needs the instance's config at ~/.config/archie/config.toml and outbound network, neither of which is a repository change. No code path is missing, stubbed or broken as far as reading shows. If the full run reports micro-recall below 0.70 or any failed problem, that report is the evidence for a follow-up code change to the review pipeline; that would be a separate item rather than this acceptance run. Note the body's reference to docs/prds/pr-review-agent.md does not resolve in this tree (PRDs for landed work are absent), but the acceptance criterion it stands for is the one implemented and documented above.
sam closed this issue 2026-11-08 22:00:13 +00:00
sam added a new dependency 2026-11-14 14:45:16 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sam/archie#131