Files
archie/docs/prds/pr-review-agent.md
T
sam d60f024e74 docs(prds): route intake's AI-score into lenses, split reviewer timeout from zero-findings, bound phase 9
Landing three PRD refinements found while studying prior reviewer art before
starting archie-core-afbk.3: dimension selection now sees intake's
machine-written score directly and adds a hallucination-check dimension for
it; a reviewer that exhausts its turn cap or deadline records a distinct
StepExecution status from one that completed and found nothing; phase 9
polish runs in a bounded pool of 8, matching phase 4.
2026-09-28 03:00:51 +10:00

210 lines
10 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# PR review agent
**Status:** Approved
**Authority:** `docs/architecture/agent-system.md`; supersedes the reviewer
executor and blocking rule in `docs/architecture/adversarial-review.md`, and
the designs in `docs/prds/operator-pr-review.md` and
`docs/prds/review-calibration.md`.
**Compounds with:** `docs/prds/execution-tree-state-machine.md` (every agent
call is a StepExecution), `docs/prds/external-agent-harness.md` (any agent
step may run on a harness), `docs/prds/inline-review.md` (posting),
`docs/prds/pr-review-remediation.md` (consumes the findings).
## Decision
Archie ships a pull request reviewer as its first end-to-end product. It
reviews any pull request on a watched repository, a PR an operator names, and
archie's own PRs before they open.
The reviewer is a fixed pipeline, not a single agent. The review strategy is
generated per PR: three lens agents propose review dimensions from the PR's
content, one reviewer definition runs once per dimension, and later agents
verify, challenge and combine the findings. Everything that can be computed
is code: diff parsing, clustering, blast radius, evidence extraction,
scoring, deduplication, line mapping and the review event.
The pipeline is **recall-first**. It posts every evidenced finding above the
confidence threshold, up to the comment cap. A precision gate exists as an
operator dial and is off by default.
Acceptance is measured, not asserted: see Verification.
## Pipeline
Each numbered phase is a stage of the `pr-review` workflow. Each agent call is
an `agent` StepExecution; fan-out phases record one child per call.
1. **Intake.** Code reads the PR metadata and diff statistics: type,
complexity, languages, areas touched, risk signals. Depth comes from lines
changed: under 100 is `quick` (at most 3 dimensions), under 500 is
`standard` (6), otherwise `deep` (12). An operator may set depth
explicitly.
2. **Anatomy.** Code parses the diff into files and hunks, clusters related
files, and computes the blast radius: unchanged files that import or are
imported by changed files. One agent writes the PR narrative, risk
surfaces, changes that do not fit the narrative, and gaps between the
description and the diff.
3. **Dimension selection.** Three lens agents run in parallel over the intake
(including its machine-written score), anatomy and diff, each able to read
the repository. A PR that intake scores as likely machine-written adds a
hallucination-check dimension: claims, citations and behaviour the diff
does not actually implement.
- **behaviour:** where the old and new code diverge for any input; API
contracts, concurrency and state, security, error handling, data flow;
- **mechanics:** whether the code runs at the language and framework
level; types, signatures and every caller, decorators and middleware,
framework contracts, imports;
- **fit:** patterns, complexity, abstraction, test adequacy, documentation
that no longer matches, dependencies, migration completeness.
Each lens returns review dimensions: a name, a reviewer prompt written for
this PR, target files, context files and a priority. Code merges them,
drops duplicates and keeps the highest-priority dimensions up to the
depth's cap.
4. **Review.** One reviewer definition runs once per dimension, in parallel,
at most 8 at a time. Each reads its target and context files and may
follow references. Each finding carries file, line range, severity
(`critical`, `important`, `suggestion`, `nitpick`), title, body, an
optional suggested fix, quoted evidence, confidence and tags. A reviewer
that exhausts its turn cap or deadline before its terminal tool call
records a distinct StepExecution status from a reviewer that completed
and found nothing; synthesis and the posted review treat the two
differently (the former is unreviewed, the latter is reviewed-clean).
5. **Verification layer**, over the review findings:
- code extracts an evidence package per finding: the code at the cited
lines and snippets from its callers;
- an evidence verifier checks high-priority findings against their
packages;
- an adversary confirms or challenges each finding, looks for issues every
reviewer missed, and is more sceptical when intake scores the PR as
likely machine-written;
- a compound agent looks, within each cluster of confirmed findings, for
defects that only exist in combination.
6. **Coverage and consistency**, in parallel:
- a coverage gate checks every cluster and high-exposure blast-radius file
was reviewed, and runs gap reviewers for what was not, at most 2 rounds;
- consistency verification lists every cross-location obligation the
changed code creates (what must be true elsewhere for this line to be
correct), then verifies each by reading both ends. A broken obligation
becomes a finding.
7. **Synthesis**, code only:
- score = severity weight (1.0, 0.7, 0.3, 0.1) × confidence × multipliers:
compound 1.5, adversary-confirmed 1.3, adversary-challenged 0.5, likely
machine-written PR 1.2, blast radius over 10 files 1.2;
- drop findings under the severity's confidence floor: critical 0.2,
important 0.3, suggestion 0.4, nitpick 0.4;
- merge exact duplicates (file, overlapping lines, category) in code, and
near-duplicates through a small classification call that keeps both
when unsure;
- map lines to the diff's coordinates, sort by score, cap at 25 inline
comments.
8. **Merge gate.** One classification call per finding decides blocking or
advisory: blocking only for broken builds, security, data loss, contract
breaks and regressions. A failed call means advisory.
9. **Output.** One call per comment tightens its wording, running in a
bounded pool at most 8 at a time (matching phase 4), keeping the original
on failure. The review is posted as one forge review with inline
comments. Any blocking finding requests changes; otherwise the review is a
comment. Archie never posts an approval.
## Precision dial
With `review.precision_gate = true`, a post-worthiness pass runs after phase 4
and before the verification layer. It keeps every concrete, evidenced defect
and drops nitpicks, style and unverifiable claims, keeping a finding when
unsure. It trades recall for precision and cuts verification cost by
filtering early.
## Operator approval
With `review.approve_before_post = true`, synthesis ends in
`waiting_human`. The operator chooses which findings to post, rejects the
review, or asks for a re-review with instructions. A re-review reruns phases
3 to 8 with the instructions added to the lens and reviewer prompts, at most
2 times. A review with no findings posts nothing and does not wait.
## Isolation
Every agent in the pipeline reads a `.git`-free snapshot of the PR head, with
read-only tools rooted at it, and never the implementer's worktree or
transcript. No pipeline agent holds a forge credential; posting is code.
## Budget
A run has a cost cap and a wall-clock cap, split into per-phase shares. A
phase whose share is spent is skipped and recorded on its StepExecution. The
posted review names every skipped phase, so an exhausted run never reads as a
clean one. A run whose review stage fails posts nothing and
fails.
## Triggers
- **Watched repositories:** a PR opened or updated on a repository with
`review.enabled` starts a run. An update cancels the in-flight run for the
same PR.
- **Operator:** a PR URL from the dashboard or chat starts a run on any
repository the identity can read.
- **Archie's own PRs:** the implement workflow runs the pipeline before
opening its PR. A blocking finding the adversary did not challenge parks
the task with the findings. A pipeline that fails parks the task. Advisory
findings are posted after the PR opens.
## Organisations
The reviewer is a workflow, so `docs/prds/orgs-and-access.md` applies to it
unchanged: it runs as the workflow's identity under the run credential, reads
only repositories that identity may read, and posts through the forge RPC
under that credential. A watched repository belongs to a workspace, and its
PR events start reviews through that workspace's bindings. Findings,
StepExecutions and the posted review carry the run's org and workspace.
## Models
Each agent phase names a model role. Lens, reviewer, verification,
adversary, compound and consistency agents use the review role. Intake,
coverage, dedup, merge gate and polish calls use the classification role.
Either role may run on the built-in runner or a harness.
## Out of scope
- SARIF and standalone report formats. The findings are StepExecution outputs
and the forge review.
- Reviewing a diff without a forge PR.
- Self-consistency reruns.
## Plan
1. Deterministic core: diff parsing, clustering, blast radius, evidence
extraction, scoring, dedup, line mapping, review event. Table-driven tests.
2. The benchmark harness (Verification), run against the deterministic core
with fixture findings.
3. `pr-review` workflow: intake, anatomy, lenses, reviewers, synthesis,
output, with the execution tree recording each call.
4. Verification layer, coverage loop and consistency verification.
5. Merge gate, polish, budget shares and skipped-phase reporting.
6. Benchmark acceptance run.
7. Triggers: operator, watched repositories, archie's own PRs replacing the
single-reviewer stage.
8. Precision dial and operator approval.
## Verification
- **Benchmark acceptance.** On the 38 runnable PRs of the Martian
Code-Review-Bench offline set, one blind run per PR at `deep` with the
review and classification roles on GLM-5.2, and an independent judge
(`anthropic/claude-sonnet-4.6`) matching posted comments to golden
comments: golden micro-recall is at least 0.70. Golden-only precision is
reported alongside it. The run is reproducible from a script in the
repository.
- Scoring, thresholds, dedup and line mapping give identical output for
identical findings.
- A run whose phase share is exhausted posts a review naming the skipped
phase.
- An update to a PR under review cancels the running review and its child
steps.
- No pipeline agent can write to the snapshot, and none can reach a forge
credential.
- Archie's own PR with an unchallenged blocking finding is parked, not
opened.
- Archie never submits an approving review.