docs(prds): route intake's AI-score into lenses, split reviewer timeout from zero-findings, bound phase 9

Landing three PRD refinements found while studying prior reviewer art before
starting archie-core-afbk.3: dimension selection now sees intake's
machine-written score directly and adds a hallucination-check dimension for
it; a reviewer that exhausts its turn cap or deadline records a distinct
StepExecution status from one that completed and found nothing; phase 9
polish runs in a bounded pool of 8, matching phase 4.
This commit is contained in:
sam committed 2026-09-28 03:00:51 +10:00
1 parent ef3cda8179
commit f20d226369
2 files changed
+14 -6

No files matched your search

+1 -1
View File
@@ -412,7 +412,7 @@
"url": "/dev/prds/pr-review-agent/",
"title": "PR review agent",
"section": "PRDs",
"body": "# PR review agent\n\n**Status:** Approved\n**Authority:** `docs/architecture/agent-system.md`; supersedes the reviewer\nexecutor and blocking rule in `docs/architecture/adversarial-review.md`, and\nthe designs in `docs/prds/operator-pr-review.md` and\n`docs/prds/review-calibration.md`.\n**Compounds with:** `docs/prds/execution-tree-state-machine.md` (every agent\ncall is a StepExecution), `docs/prds/external-agent-harness.md` (any agent\nstep may run on a harness), `docs/prds/inline-review.md` (posting),\n`docs/prds/pr-review-remediation.md` (consumes the findings).\n\n## Decision\n\nArchie ships a pull request reviewer as its first end-to-end product. It\nreviews any pull request on a watched repository, a PR an operator names, and\narchie's own PRs before they open.\n\nThe reviewer is a fixed pipeline, not a single agent. The review strategy is\ngenerated per PR: three lens agents propose review dimensions from the PR's\ncontent, one reviewer definition runs once per dimension, and later agents\nverify, challenge and combine the findings. Everything that can be computed\nis code: diff parsing, clustering, blast radius, evidence extraction,\nscoring, deduplication, line mapping and the review event.\n\nThe pipeline is **recall-first**. It posts every evidenced finding above the\nconfidence threshold, up to the comment cap. A precision gate exists as an\noperator dial and is off by default.\n\nAcceptance is measured, not asserted: see Verification.\n\n## Pipeline\n\nEach numbered phase is a stage of the `pr-review` workflow. Each agent call is\nan `agent` StepExecution; fan-out phases record one child per call.\n\n1. **Intake.** Code reads the PR metadata and diff statistics: type,\n complexity, languages, areas touched, risk signals. Depth comes from lines\n changed: under 100 is `quick` (at most 3 dimensions), under 500 is\n `standard` (6), otherwise `deep` (12). An operator may set depth\n explicitly.\n2. **Anatomy.** Code parses the diff into files and hunks, clusters related\n files, and computes the blast radius: unchanged files that import or are\n imported by changed files. One agent writes the PR narrative, risk\n surfaces, changes that do not fit the narrative, and gaps between the\n description and the diff.\n3. **Dimension selection.** Three lens agents run in parallel over the intake,\n anatomy and diff, each able to read the repository:\n - **behaviour:** where the old and new code diverge for any input; API\n contracts, concurrency and state, security, error handling, data flow;\n - **mechanics:** whether the code runs at the language and framework\n level; types, signatures and every caller, decorators and middleware,\n framework contracts, imports;\n - **fit:** patterns, complexity, abstraction, test adequacy, documentation\n that no longer matches, dependencies, migration completeness.\n\n Each lens returns review dimensions: a name, a reviewer prompt written for\n this PR, target files, context files and a priority. Code merges them,\n drops duplicates and keeps the highest-priority dimensions up to the\n depth's cap.\n4. **Review.** One reviewer definition runs once per dimension, in parallel,\n at most 8 at a time. Each reads its target and context files and may\n follow references. Each finding carries file, line range, severity\n (`critical`, `important`, `suggestion`, `nitpick`), title, body, an\n optional suggested fix, quoted evidence, confidence and tags.\n5. **Verification layer**, over the review findings:\n - code extracts an evidence package per finding: the code at the cited\n lines and snippets from its callers;\n - an evidence verifier checks high-priority findings against their\n packages;\n - an adversary confirms or challenges each finding, looks for issues every\n reviewer missed, and is more sceptical when intake scores the PR as\n likely machine-written;\n - a compound agent looks, within each cluster of confirmed findings, for\n defects that only exist in combination.\n6. **Coverage and consistency**, in parallel:\n - a coverage gate checks every cluster and high-exposure blast-radius file\n was reviewed, and runs gap reviewers for what was not, at most 2 rounds;\n - consistency verification lists every cross-location obligation the\n changed code creates (what must be true elsewhere for this line to be\n correct), then verifies each by reading both ends. A broken obligation\n becomes a finding.\n7. **Synthesis**, code only:\n - score = severity weight (1.0, 0.7, 0.3, 0.1) × confidence × multipliers:\n compound 1.5, adversary-confirmed 1.3, adversary-challenged 0.5, likely\n machine-written PR 1.2, blast radius over 10 files 1.2;\n - drop findings under the severity's confidence floor: critical 0.2,\n important 0.3, suggestion 0.4, nitpick 0.4;\n - merge exact duplicates (file, overlapping lines, category) in code, and\n nearLine truncated
"body": "# PR review agent\n\n**Status:** Approved\n**Authority:** `docs/architecture/agent-system.md`; supersedes the reviewer\nexecutor and blocking rule in `docs/architecture/adversarial-review.md`, and\nthe designs in `docs/prds/operator-pr-review.md` and\n`docs/prds/review-calibration.md`.\n**Compounds with:** `docs/prds/execution-tree-state-machine.md` (every agent\ncall is a StepExecution), `docs/prds/external-agent-harness.md` (any agent\nstep may run on a harness), `docs/prds/inline-review.md` (posting),\n`docs/prds/pr-review-remediation.md` (consumes the findings).\n\n## Decision\n\nArchie ships a pull request reviewer as its first end-to-end product. It\nreviews any pull request on a watched repository, a PR an operator names, and\narchie's own PRs before they open.\n\nThe reviewer is a fixed pipeline, not a single agent. The review strategy is\ngenerated per PR: three lens agents propose review dimensions from the PR's\ncontent, one reviewer definition runs once per dimension, and later agents\nverify, challenge and combine the findings. Everything that can be computed\nis code: diff parsing, clustering, blast radius, evidence extraction,\nscoring, deduplication, line mapping and the review event.\n\nThe pipeline is **recall-first**. It posts every evidenced finding above the\nconfidence threshold, up to the comment cap. A precision gate exists as an\noperator dial and is off by default.\n\nAcceptance is measured, not asserted: see Verification.\n\n## Pipeline\n\nEach numbered phase is a stage of the `pr-review` workflow. Each agent call is\nan `agent` StepExecution; fan-out phases record one child per call.\n\n1. **Intake.** Code reads the PR metadata and diff statistics: type,\n complexity, languages, areas touched, risk signals. Depth comes from lines\n changed: under 100 is `quick` (at most 3 dimensions), under 500 is\n `standard` (6), otherwise `deep` (12). An operator may set depth\n explicitly.\n2. **Anatomy.** Code parses the diff into files and hunks, clusters related\n files, and computes the blast radius: unchanged files that import or are\n imported by changed files. One agent writes the PR narrative, risk\n surfaces, changes that do not fit the narrative, and gaps between the\n description and the diff.\n3. **Dimension selection.** Three lens agents run in parallel over the intake\n (including its machine-written score), anatomy and diff, each able to read\n the repository. A PR that intake scores as likely machine-written adds a\n hallucination-check dimension: claims, citations and behaviour the diff\n does not actually implement.\n - **behaviour:** where the old and new code diverge for any input; API\n contracts, concurrency and state, security, error handling, data flow;\n - **mechanics:** whether the code runs at the language and framework\n level; types, signatures and every caller, decorators and middleware,\n framework contracts, imports;\n - **fit:** patterns, complexity, abstraction, test adequacy, documentation\n that no longer matches, dependencies, migration completeness.\n\n Each lens returns review dimensions: a name, a reviewer prompt written for\n this PR, target files, context files and a priority. Code merges them,\n drops duplicates and keeps the highest-priority dimensions up to the\n depth's cap.\n4. **Review.** One reviewer definition runs once per dimension, in parallel,\n at most 8 at a time. Each reads its target and context files and may\n follow references. Each finding carries file, line range, severity\n (`critical`, `important`, `suggestion`, `nitpick`), title, body, an\n optional suggested fix, quoted evidence, confidence and tags. A reviewer\n that exhausts its turn cap or deadline before its terminal tool call\n records a distinct StepExecution status from a reviewer that completed\n and found nothing; synthesis and the posted review treat the two\n differently (the former is unreviewed, the latter is reviewed-clean).\n5. **Verification layer**, over the review findings:\n - code extracts an evidence package per finding: the code at the cited\n lines and snippets from its callers;\n - an evidence verifier checks high-priority findings against their\n packages;\n - an adversary confirms or challenges each finding, looks for issues every\n reviewer missed, and is more sceptical when intake scores the PR as\n likely machine-written;\n - a compound agent looks, within each cluster of confirmed findings, for\n defects that only exist in combination.\n6. **Coverage and consistency**, in parallel:\n - a coverage gate checks every cluster and high-exposure blast-radius file\n was reviewed, and runs gap reviewers for what was not, at most 2 rounds;\n - consistency verification lists every cross-location obligation the\n changed code creates (what must be true elsewhere for this line to be\n correct), then verifies each by reading both endLine truncated
},
{
"url": "/dev/prds/pr-review-remediation/",
+13 -5
View File
@@ -44,8 +44,11 @@ an `agent` StepExecution; fan-out phases record one child per call.
imported by changed files. One agent writes the PR narrative, risk
surfaces, changes that do not fit the narrative, and gaps between the
description and the diff.
3. **Dimension selection.** Three lens agents run in parallel over the intake,
anatomy and diff, each able to read the repository:
3. **Dimension selection.** Three lens agents run in parallel over the intake
(including its machine-written score), anatomy and diff, each able to read
the repository. A PR that intake scores as likely machine-written adds a
hallucination-check dimension: claims, citations and behaviour the diff
does not actually implement.
- **behaviour:** where the old and new code diverge for any input; API
contracts, concurrency and state, security, error handling, data flow;
- **mechanics:** whether the code runs at the language and framework
@@ -62,7 +65,11 @@ an `agent` StepExecution; fan-out phases record one child per call.
at most 8 at a time. Each reads its target and context files and may
follow references. Each finding carries file, line range, severity
(`critical`, `important`, `suggestion`, `nitpick`), title, body, an
optional suggested fix, quoted evidence, confidence and tags.
optional suggested fix, quoted evidence, confidence and tags. A reviewer
that exhausts its turn cap or deadline before its terminal tool call
records a distinct StepExecution status from a reviewer that completed
and found nothing; synthesis and the posted review treat the two
differently (the former is unreviewed, the latter is reviewed-clean).
5. **Verification layer**, over the review findings:
- code extracts an evidence package per finding: the code at the cited
lines and snippets from its callers;
@@ -94,8 +101,9 @@ an `agent` StepExecution; fan-out phases record one child per call.
8. **Merge gate.** One classification call per finding decides blocking or
advisory: blocking only for broken builds, security, data loss, contract
breaks and regressions. A failed call means advisory.
9. **Output.** One call per comment tightens its wording, keeping the
original on failure. The review is posted as one forge review with inline
9. **Output.** One call per comment tightens its wording, running in a
bounded pool at most 8 at a time (matching phase 4), keeping the original
on failure. The review is posted as one forge review with inline
comments. Any blocking finding requests changes; otherwise the review is a
comment. Archie never posts an approval.