Golden micro-recall at least 0.70. See docs/prds/pr-review-agent.md.
Imported from beads archie-core-afbk.6 · GitHub #1199 · type task · priority P1 · created 2026-09-25
Golden micro-recall at least 0.70. See docs/prds/pr-review-agent.md.
---
_Imported from beads `archie-core-afbk.6` · GitHub #1199 · type task · priority P1 · created 2026-09-25_
Triage 2026-10-02 @ 11974196: PARTIAL. Harness + metric + gate and external-PR sourcing are SHIPPED (1c7dd08f harness, bae53bcb live pipeline, dabe7c88 current form; prsource 7ddaed38, daemon prefetch 2142e446). The bead's own acceptance run has NOT happened: zero acceptance artifacts exist. REMAINING: run the 38-PR one-blind-deep acceptance (task bench:prreview) and record recall >= 0.70 durably -- bin/ is gitignored, so a summary.json there is not tree evidence. BLOCKER is model availability, not code: the PRD names GLM-5.2 for review+classification and anthropic/claude-sonnet-4.6 as judge, and substitutions do not satisfy the criterion (docs/guides/pr-review-benchmark.md:28). Needs GH_TOKEN with read access to the 38 upstream Martian PRs. This bead gates em7s.8.
_sam wrote (originally 2026-10-02):_
Triage 2026-10-02 @ 11974196: PARTIAL. Harness + metric + gate and external-PR sourcing are SHIPPED (1c7dd08f harness, bae53bcb live pipeline, dabe7c88 current form; prsource 7ddaed38, daemon prefetch 2142e446). The bead's own acceptance run has NOT happened: zero acceptance artifacts exist. REMAINING: run the 38-PR one-blind-deep acceptance (task bench:prreview) and record recall >= 0.70 durably -- bin/ is gitignored, so a summary.json there is not tree evidence. BLOCKER is model availability, not code: the PRD names GLM-5.2 for review+classification and anthropic/claude-sonnet-4.6 as judge, and substitutions do not satisfy the criterion (docs/guides/pr-review-benchmark.md:28). Needs GH_TOKEN with read access to the 38 upstream Martian PRs. This bead gates em7s.8.
Runner restored (commit 5f077d20 on branch worktree-agent-ab430d62419672912, unmerged). The old runner and task had been deleted as orphaned. To run the acceptance: set the default model alias in ~/.config/archie/config.toml (or pass flags), then 'task bench:prreview -- -model ALIAS_OR_PROVIDER/MODEL -judge-model ALIAS_OR_PROVIDER/MODEL'. No GH_TOKEN needed (public PRs; token only lifts API rate limit). Commit the docs/benchmarks/pr-review-*.md report it writes. See docs/guides/pr-review-benchmark.md. Acceptance run itself is not done.
_sam wrote (originally 2026-10-10):_
Runner restored (commit 5f077d20 on branch worktree-agent-ab430d62419672912, unmerged). The old runner and task had been deleted as orphaned. To run the acceptance: set the default model alias in ~/.config/archie/config.toml (or pass flags), then 'task bench:prreview -- -model ALIAS_OR_PROVIDER/MODEL -judge-model ALIAS_OR_PROVIDER/MODEL'. No GH_TOKEN needed (public PRs; token only lifts API rate limit). Commit the docs/benchmarks/pr-review-*.md report it writes. See docs/guides/pr-review-benchmark.md. Acceptance run itself is not done.
sam
added a new dependency 2026-10-27 13:41:45 +00:00
Triaged: no code change is needed for this. This is an acceptance run, not a build. The benchmark harness already exists and is complete: task bench:prreview runs archied prbench (internal/app/prbench), which reviews the 38 embedded Martian PRs in internal/app/prbench/martian.json with the production pipeline, judges posted comments against the goldens, and writes docs/benchmarks/pr-review-.md. The 0.70 golden micro-recall bar from the PRD is implemented as acceptanceRecall in internal/app/prbench/prbench.go, and a full run reports PASS/FAIL against it; docs/guides/pr-review-benchmark.md documents the command, flags and acceptance rule.
What remains is to execute that command with the operator's configured models (the reviewer and an independent judge) and commit the generated report; that needs the instance's config at ~/.config/archie/config.toml and outbound network, neither of which is a repository change. No code path is missing, stubbed or broken as far as reading shows.
If the full run reports micro-recall below 0.70 or any failed problem, that report is the evidence for a follow-up code change to the review pipeline; that would be a separate item rather than this acceptance run. Note the body's reference to docs/prds/pr-review-agent.md does not resolve in this tree (PRDs for landed work are absent), but the acceptance criterion it stands for is the one implemented and documented above.
Triaged: no code change is needed for this. This is an acceptance run, not a build. The benchmark harness already exists and is complete: `task bench:prreview` runs `archied prbench` (internal/app/prbench), which reviews the 38 embedded Martian PRs in internal/app/prbench/martian.json with the production pipeline, judges posted comments against the goldens, and writes docs/benchmarks/pr-review-<timestamp>.md. The 0.70 golden micro-recall bar from the PRD is implemented as `acceptanceRecall` in internal/app/prbench/prbench.go, and a full run reports PASS/FAIL against it; docs/guides/pr-review-benchmark.md documents the command, flags and acceptance rule.
What remains is to execute that command with the operator's configured models (the reviewer and an independent judge) and commit the generated report; that needs the instance's config at ~/.config/archie/config.toml and outbound network, neither of which is a repository change. No code path is missing, stubbed or broken as far as reading shows.
If the full run reports micro-recall below 0.70 or any failed problem, that report is the evidence for a follow-up code change to the review pipeline; that would be a separate item rather than this acceptance run. Note the body's reference to docs/prds/pr-review-agent.md does not resolve in this tree (PRDs for landed work are absent), but the acceptance criterion it stands for is the one implemented and documented above.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Golden micro-recall at least 0.70. See docs/prds/pr-review-agent.md.
Imported from beads
archie-core-afbk.6· GitHub #1199 · type task · priority P1 · created 2026-09-25sam wrote (originally 2026-10-02):
Triage 2026-10-02 @
11974196: PARTIAL. Harness + metric + gate and external-PR sourcing are SHIPPED (1c7dd08fharness,bae53bcblive pipeline,dabe7c88current form; prsource7ddaed38, daemon prefetch2142e446). The bead's own acceptance run has NOT happened: zero acceptance artifacts exist. REMAINING: run the 38-PR one-blind-deep acceptance (task bench:prreview) and record recall >= 0.70 durably -- bin/ is gitignored, so a summary.json there is not tree evidence. BLOCKER is model availability, not code: the PRD names GLM-5.2 for review+classification and anthropic/claude-sonnet-4.6 as judge, and substitutions do not satisfy the criterion (docs/guides/pr-review-benchmark.md:28). Needs GH_TOKEN with read access to the 38 upstream Martian PRs. This bead gates em7s.8.sam wrote (originally 2026-10-10):
Runner restored (commit 5f077d20 on branch worktree-agent-ab430d62419672912, unmerged). The old runner and task had been deleted as orphaned. To run the acceptance: set the default model alias in ~/.config/archie/config.toml (or pass flags), then 'task bench:prreview -- -model ALIAS_OR_PROVIDER/MODEL -judge-model ALIAS_OR_PROVIDER/MODEL'. No GH_TOKEN needed (public PRs; token only lifts API rate limit). Commit the docs/benchmarks/pr-review-*.md report it writes. See docs/guides/pr-review-benchmark.md. Acceptance run itself is not done.
Triaged: no code change is needed for this. This is an acceptance run, not a build. The benchmark harness already exists and is complete:
task bench:prreviewrunsarchied prbench(internal/app/prbench), which reviews the 38 embedded Martian PRs in internal/app/prbench/martian.json with the production pipeline, judges posted comments against the goldens, and writes docs/benchmarks/pr-review-.md. The 0.70 golden micro-recall bar from the PRD is implemented asacceptanceRecallin internal/app/prbench/prbench.go, and a full run reports PASS/FAIL against it; docs/guides/pr-review-benchmark.md documents the command, flags and acceptance rule.What remains is to execute that command with the operator's configured models (the reviewer and an independent judge) and commit the generated report; that needs the instance's config at ~/.config/archie/config.toml and outbound network, neither of which is a repository change. No code path is missing, stubbed or broken as far as reading shows.
If the full run reports micro-recall below 0.70 or any failed problem, that report is the evidence for a follow-up code change to the review pipeline; that would be a separate item rather than this acceptance run. Note the body's reference to docs/prds/pr-review-agent.md does not resolve in this tree (PRDs for landed work are absent), but the acceptance criterion it stands for is the one implemented and documented above.