feat(docs): emit the canonical public-docs artifact the app renders

MkDocs retires and the Astro app becomes the only public site, so the artifact is
the product and the Markdown stays the source -- the split tools/newsgen already
states for releases. tools/docsite walks the published set and writes
docs/data/generated/docs.json: every page's directory-style url, title, section and
raw Markdown body, plus the sidebar tree derived from the layout.

Three things the artifact has to get right, each of which was wrong in a draft:

- LINKS. The published pages carry relative .md links that MkDocs rewrote at build
  time and nothing else will. Each is rewritten to the target page's url using the
  same function that builds urls, so a link and its target cannot disagree; anchors
  and scheme-qualified links pass through; assets resolve by the same rule and none
  are referenced.
- FRAGMENTS. A link can resolve to a page and still be broken, because a fragment
  matching no heading lands the reader at the top of a long document. Each is
  checked against the target's headings, by the slug rule a table of contents uses.
- REFUSAL. A link whose target the set holds back, or whose fragment matches no
  heading, fails the generator naming the file, the line and the target. This found
  three citations from settled architecture pointing at two PRDs the blanket prds/*
  exclusion withheld -- links that were already 404s on the published site, so the
  exclusion broke citations rather than withholding drafts. Both targets are now
  published, for a reason distinct from the three ratified PRDs already published.

The published set moves out of mkdocs.yml into the generator with its reasons: a
renderer reading the artifact cannot see why a page is absent. docs/data/generated
is accounted for by docsgen's generatedArtifacts, which now names this file beside
contracts.json -- one ownership registry for the directory rather than one per
writer, so neither tool reports the other's artifact as unaccounted.

docs:artifact regenerates; docs:artifact:check reproduces into memory and fails on
drift without touching the tree, wired into task check beside news:check. The
existing docs:site target is MkDocs and was left alone -- a first attempt at that
name silently ran MkDocs and produced nothing.

Verified: task check exit 0, docs:artifact:check reporting 23 pages. The artifact
carries 23 pages, no archive/ inspiration/ or news/ entry, and the five PRDs the
set declares. Coverage: the published set and its count, url derivation, the nav
tree, both refusal paths, and check's no-write guarantee.
This commit is contained in:
sam committed 2026-09-22 12:44:18 +10:00
1 parent fa517fb49e
commit 1b54a18e43
7 files changed
+1168 -24

No files matched your search

+38
View File
@@ -131,6 +131,41 @@ tasks:
cmds:
- go -C tools run -mod=readonly ./newsgen check --repo-root ..
docs:artifact:
desc: Regenerate the canonical public-docs artifact from the Markdown sources
summary: |
Writes docs/data/generated/docs.json: every published page's url, title,
section and raw Markdown body, plus the sidebar tree derived from the
layout. The JSON is the canonical, host-neutral fact set a renderer reads;
the Markdown stays the source. MkDocs was the first renderer and the Astro
app is the second, so the artifact is the product.
The published set lives in tools/docsite/main.go with the reasons for what
is held back -- archive/, inspiration/, news/, most of prds/ and the
credential notes -- because a renderer reading the artifact cannot see why
a page is absent.
The file is committed, so this dirties the tree deliberately: review the
diff and commit it with the page that produced it.
The artifact's keys are a frozen contract: an application is built against
them, so renaming one breaks a consumer this repository does not compile.
cmds:
- go -C tools run -mod=readonly ./docsite --repo-root ..
docs:artifact:check:
desc: Verify the committed public-docs artifact matches the Markdown sources
summary: |
Reproduces the artifact in a temporary directory and fails when the
committed file differs, without modifying the working tree. Runs in
`task check`, so a page edit cannot land without its regeneration.
The generated directory is accounted for by docsgen's generatedArtifacts,
which names this file alongside contracts.json: one registry for the
directory rather than one per writer.
cmds:
- go -C tools run -mod=readonly ./docsite check --repo-root ..
vet:
desc: Run go vet
cmds:
@@ -454,6 +489,9 @@ tasks:
# already freezes. Left out of the gate, a changelog edit would land
# while the published notes silently lag behind it.
- task: news:check
# The public-docs artifact is what the site renders, so a page edit that
# does not reach it publishes a site that disagrees with the repository.
- task: docs:artifact:check
- task: vet
- task: lint
- task: build
+4 -1
View File
@@ -140,6 +140,7 @@ docs/
generated/
catalog.json
contracts.json
docs.json
cli.json
configuration.json
messaging.json
@@ -345,7 +346,9 @@ go -C tools run -mod=readonly ./docsgen check --repo-root ..
```
`docsgen all` is still target-only: `docsgen data` and `docsgen asyncapi` do not
exist, and only the single `contracts.json` artifact is generated today.
exist. Two artifacts are generated today: `contracts.json` from the Go
contract types (`docsgen`) and `docs.json` from the Markdown sources
(`docsite`, the published set a renderer reads) -- see `task docs:artifact`.
### Step 9: CI
+248
View File
@@ -0,0 +1,248 @@
{
"pages": [
{
"url": "/docs/",
"title": "Archie",
"section": "",
"body": "# Archie\n\nEvent Driven Agentic Automation.\n\n**archied** is a resident daemon. It polls a forge (GitHub, Gitea) for issues\nassigned or labelled to it, works each one in an isolated git worktree through a\nrouted workflow, and opens a pull request for human review.\n\nWork arrives as events. Each task runs in its own sandbox with a task-scoped\ncredential, and the daemon performs all git operations.\n\n## Start here\n\n- [First playbook](/docs/guides/first-playbook/) walks through getting one piece of\n work from an issue to a pull request.\n- [Architecture](/docs/architecture/) is the authority for settled design:\n the package map, the workflow engine, the task lifecycle, and the invariants\n that hold the whole thing together.\n\n## What runs where\n\nA deployment is one daemon plus the processes it talks to. The State Store is a\nstandalone process and owns the database outright; the daemon and gateway never\nopen it directly. Agents execute in containers that receive a task-scoped\ncredential, issued per task and revoked when the worktree is handed back.\n\nConfiguration lives at `${XDG_CONFIG_HOME:-~/.config}/archie/config.toml`.\nNothing in a working tree is load-bearing at runtime.\n\n## A note on these docs\n\nThis site is generated from the `docs/` directory of the\n[archie-core repository](https://github.com/samcharles93/archie-core) on every\nchange to `main`. The repository is the source; this site is a view of it. If\nthe two ever disagree, the repository is right.\n"
},
{
"url": "/docs/architecture/",
"title": "Architecture",
"section": "Architecture",
"body": "# Architecture\n\nArchie is a personal agent platform built around six domain boundaries:\n\n- Identity tracks users and other actors responsible for actions.\n- Agent owns persistent assistants, specialisation, and continuity.\n- Messaging owns conversations, messages, branches, and channel correlation.\n- Workflow owns reusable, versioned behaviour definitions and their executions.\n- Work Intake admits optional durable work without constraining ordinary agent\n conversation or direct capability use.\n- Plugin owns generic extension discovery and lifecycle; each capability family\n owns its typed plugin contract.\n\nThe detailed architecture and migration decisions are maintained under\n`docs/architecture/` in the repository while this generated site is being\nassembled.\n"
},
{
"url": "/docs/architecture/adversarial-review/",
"title": "Adversarial self-review",
"section": "Architecture",
"body": "# Adversarial self-review\n\nArchie runs a fresh adversarial review of the diff before it opens a pull\nrequest, so a PR arrives with a findings record rather than an unreviewed\nclaim. The reviewer provably cannot see the implementer's context: it reviews a\n`.git`-free snapshot of the committed tree, not the worktree, the commit\nhistory, or the implementer's transcript.\n\nPromoted from `docs/prds/adversarial-self-review.md` (the design decision;\n`archie-core-h019`). Describes the stage as shipped.\n\n## Contract (`internal/domain/workflow`)\n\nThe workflow domain owns the contract and the blocking decision; it never names\nthe executor.\n\n- `Reviewer.Review(ctx, ReviewRequest) ReviewReport` — the interface the\n workflow stage consumes.\n- `ReviewRequest` is the reviewer's entire input: `SnapshotDir` (a `.git`-free\n export of the reviewed commit), `Diff` (base...head), `IssueText` (the\n originating issue title/body), and `MaxSteps`.\n- `ReviewFinding` / `ReviewReport` are the structured result. The load-bearing\n rule is `Blocking() == (Verdict == confirmed \u0026\u0026 Level == error)`: only a\n confirmed error-level finding blocks; a plausible finding never blocks\n however severe it claims to be.\n\n`ReviewReport` distinguishes `not_run`, `skipped`, and `completed` (zero\nfindings) structurally, so \"ran and found nothing\" is not the same as \"never\nran\" — collapsing them would make an outage indistinguishable from a clean\nreview.\n\n## Execution (`internal/app/agentworker`)\n\nThe executor is an ai-sdk `agent.Subagent`: a nested synchronous generation in\nthe same worker process, with its own model, system prompt, toolset and step\nbudget.\n\n- **Conversation isolation is structural.** `Subagent.Run` sends only a prompt\n string, never the implementer's message history.\n- **Workspace isolation is the `.git`-free snapshot.** The reviewer's toolset is\n read-only (`read`, `grep`, `find`) and rooted at `SnapshotDir` — never the\n worker's own toolset, which carries the `worktreerpc` publication grant.\n- **Findings return through a typed capture tool** (`record_finding`), validated\n on the way in, so the reviewer reports through the contract rather than free\n text the caller regexes.\n- **Fail-closed.** A reviewer that errors, truncates, or reaches no conclusion\n yields `ReviewStatusNotRun`, which the stage treats as a park — never a silent\n \"PR opened unreviewed\".\n\n## Lifecycle effect\n\n`StageReview` runs in the implement workflow after the deterministic gates and\nbefore `StageOpenPR`, gated by `repo.review_enabled` (opt-in, default off). A\nsurviving blocking finding — or a review that failed to run — parks the task\n(`StatusParked`, not `waiting_human`) with the findings rendered into the park\nreason; `StageOpenPR` never runs. Warn/plausible findings and the zero-finding\ncase leave the outcome unset, so the PR opens with a findings section rendered\nonto its body. `StagePostReviewComments` then runs after `StageOpenPR` and posts\nthe **line-anchored** findings as inline review comments, carrying a one-click\nsuggestion on a confirmed finding the reviewer could state a fix for. That stage\nis best-effort — a comment that fails to post is logged, never a park — because\nthe body section already lists every finding, and it is the record a whole-file\nfinding (no line to anchor to) only ever appears in.\n\n## Model and budget\n\nThe reviewer model is `models.reviewer`, falling back to `models.builder`.\nEffort is the per-stage `MaxSteps` budget, mapped onto `Subagent.MaxSteps`\n(default 10).\n\n## Operator-triggered review\n\nThe same reviewer runs against an *existing* PR via the channel-neutral\n`review_pr` chat command: fetch the PR's head, snapshot it, run the isolated\nreviewer, and return the structured findings. Authorization is identity-scoped\nto the operator's configured repositories; duplicate or concurrent reviews of\nthe same PR are deduplicated.\n"
},
{
"url": "/docs/architecture/agent-system/",
"title": "Agent System",
"section": "Architecture",
"body": "# Agent System\n\n**Status:** Current-state investigation in progress \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Domain framing\n\nThe domain is the Agent System.\n\n`Task` is not a core Agent System concept. Forge tasks, forge issues, Jira\nissues, email, Telegram messages, webhooks, and future external inputs are\nchannel-specific messages. Channels translate those messages into requests to\nthe Agent System; external issue or task models MUST NOT shape execution state.\n\nThe current `store.Task` is a persisted workflow execution. Treating that row as\na task-domain aggregate would incorrectly turn Archie into an agentic issue\nboard.\n\n## Canonical agent interaction model\n\nAn **Agent** is a persistent assistant selected for an interaction. It is not a\nmodel invocation and is not a WorkflowExecution.\n\nEach Agent has its own:\n\n- immutable acting `IdentityID` for ownership and audit;\n- durable memory and learned information about the user;\n- perspective, specialisation, instructions, and communication behaviour;\n- conversation history and channel bindings;\n- available tools, capabilities, and policy;\n- continuity across model or provider changes.\n\nThe underlying model is a replaceable runtime dependency. Changing a model,\nprovider, or model version MUST NOT create a new Agent, discard its memory, or\nchange ownership of its history.\n\nA user MAY create multiple Agents. Each Agent develops its own memory and view\nof the user from the interactions and responsibilities assigned to it. Memories\nMUST NOT be silently pooled merely because Agents belong to the same user or\napplication.\n\nAgent and Identity are separate aggregates. Every Agent references exactly one\nacting IdentityID, and no two Agents share an agent identity. User, system, and\nservice-account identities may exist without an Agent.\n\nAn action initiated through an Agent records the Agent's acting IdentityID and,\nwhen applicable, the initiating user's IdentityID. A future fully automated\nWorkflowExecution MAY use an explicitly selected service-account identity for a\nconcrete unattended use case; this is not the default execution identity.\n\n## Workflow execution identity\n\nA Workflow is owned by an IdentityID. By default, every WorkflowExecution acts\nas the Workflow owner's identity.\n\nFuture delegated execution MAY add an explicit `RunAsIdentityID` assignment to a\nWorkflow. This allows a Workflow owner to select another user or service-account\nidentity for execution without changing Workflow ownership. Until that\ncapability is deliberately introduced, owner identity and execution identity are\nthe same.\n\nWorkflowExecution audit records MUST preserve the Workflow owner, effective\nexecution identity, initiating user when one exists, and any explicit delegation\nthat selected a different run-as identity.\n\n## Memory scopes\n\nThe Agent System requires four distinct durable memory scopes:\n\n- **global shared memory**: information intentionally available across the\n Archie application;\n- **agent-wide memory**: general expertise, operating knowledge, and perspective\n owned by one Agent and safe for that Agent to use across users;\n- **user-wide memory**: information associated with one user and available to\n authorized Agents serving that user;\n- **Agent-user relationship memory**: private history and learned context from\n one particular Agent's relationship with one particular user.\n\nAgent-wide memory is private to its owning Agent by default. Another Agent MUST\nNOT read, inherit, search, or mutate it implicitly.\n\nAgent-user relationship memory is addressed by both AgentID and User IdentityID.\nNeither another Agent serving the same user nor another user speaking to the\nsame Agent may access it implicitly.\n\nArchie is a multi-user system even when initially deployed for one person.\nMultiple users, including family members, may interact with the same configured\nAgents. Per-user memory is therefore an isolation and personalization boundary,\nnot merely an attribution tag on the primary user's records.\n\nAn Agent MUST resolve the initiating user before assembling memory context.\nMemory belonging to one user MUST NOT become visible to another user merely\nbecause both users interact with the same Agent.\n\nShared memory does not erase provenance. Every memory record retains its scope,\nauthor or acting identity, originating user when applicable, source, and\nrevision history. The rules for selecting a write scope, granting access\nto global and user-wide memory, and authorizing explicit cross-scope sharing are\nsettled by [`docs/prds/memory-engine-unification.md`](/docs/prds/memory-engine-unification/)\nand recorded in [`migration-decisions.md`](/docs/architecture/migration-decisions/#5-memory-placement-and-storage)\nsection 5.\n\nAn Agent interaction may answer conversationally, use a tool or capability, read\nLine truncated
},
{
"url": "/docs/architecture/bindings/",
"title": "Playbook bindings: model and threat model",
"section": "Architecture",
"body": "# Playbook bindings: model and threat model\n\n**Status:** Approved foundation (documents shipped code: `archie-core-t2db.1`\nthrough `.5`, `.7`, `.9`-`.15`)\n**Date:** 2026-09-03\n\nThis document covers the binding model and the intake threat model portion\nof `archie-core-t2db.6`. It does not cover the `ARCHITECTURE.md` task-lifecycle\naddendum or the operator walkthrough -- those remain separate, open pieces of\nthe same ticket.\n\n## What a binding is\n\nA `binding` (`internal/domain/binding.Binding`) ties four things together:\na **matcher** (which captured events it applies to), a **mapping**\n(`internal/domain/mapping.Mapping`, how fields are pulled out of a payload),\na **workflow** (which registered workflow runs on a match), and a **state\nmachine** governing whether it is live.\n\n```go\ntype Binding struct {\n ID, Name // identity\n Matcher // { Source string } -- the webhook path segment, e.g. \"sentry\"\n MappingID // which Mapping resolves payload fields\n Workflow // registered workflow name to dispatch to\n Version // bumped on every edit\n Status // draft | pending_approval | armed\n Secret // HMAC-SHA256 shared secret, encrypted at rest, never returned by GET\n}\n```\n\nBindings live in the store (SQLite), not `config.toml` -- operators author\nthem from the dashboard while the daemon runs, without a restart.\n\n### Matching\n\n`Matcher.Source` is the only predicate today: the path segment a sender\nPOSTs to (e.g. a webhook hitting `.../sentry` matches bindings with\n`Matcher.Source == \"sentry\"`). The storage column is plain `TEXT`, so adding\npredicates later is non-breaking -- extend `Matcher`, don't invent a second\nmatcher shape.\n\n### Mapping\n\n`mapping.Resolve(fields, payload)` walks a captured JSON payload against a\nlist of typed `Field`s (name, JSON path, expected type), returning resolved\nvalues plus a `[]Failure` for anything that didn't parse or type-match. This\nis schema-by-example in practice: a mapping is authored by pointing at a\nreal captured payload in the dashboard, not by guessing a schema in advance.\n\n### Versioning\n\n`Version` increments on every `UpdateBinding` call. This exists so an\neditor's later change can never silently rewrite the historical provenance\nof a task that already fired -- a dispatched task's binding reference is\npinned to the version that was armed when it fired, not \"whatever the\nbinding currently says.\"\n\n### State machine\n\n```\ndraft ---\u003e pending_approval ---\u003e armed\n ^ ^ |\n | | |\n +--------------+-----------------+\n (any edit to an armed binding drops it back to pending_approval)\n```\n\nModeled directly on Telegram's existing `dangerousAction`/`pendingApproval`\nflow (`internal/channels/telegram/dangerous.go`, `approval.go`) -- not a new\napproval mechanism invented for this feature. **Only `armed` evaluates\nagainst incoming events** (`binding.Matches`, `internal/domain/binding/\nbinding.go`); `draft` and `pending_approval` bindings are inert. An edit to\nan already-armed binding cannot bypass approval by silent mutation -- it\ndrops back to `pending_approval` and requires an explicit `Approve` call to\nre-arm.\n\n## The threat model\n\nThis is a public, unauthenticated-by-default intake surface whose output is\nan agent with a repo, a worktree, and tool access. The design brief\n(`archie-core-t2db.5`, `docs/prds/webhook-intake-security.md`) named five\nnon-negotiables; each is verified below against the actual shipped code,\nnot the design intent.\n\n### 1. Per-source authentication\n\nReuses `internal/channels/webhook`'s existing HMAC-SHA256 scheme --\ncapture accepts unsigned events (visible in the dashboard, marked\nunauthenticated), but **only authenticated events can ever match a\nbinding**. `binding.Matches`' doc comment states this plainly: \"an\nunauthenticated event can never trigger a binding, no matter how well it\nwould otherwise match.\" Enforced structurally at two points, not just\nconvention:\n\n- `captured_events.authenticated` is a real column (`internal/store/\n captures.go`), set at capture time.\n- `dispatchBindings` (`internal/daemon/daemon.go:479`) walks **authenticated\n captures whose source has at least one armed binding** -- an\n unauthenticated capture is never even considered for dispatch.\n\n### 2. Human approval before a binding goes live\n\nCovered by the state machine above. Nothing self-arms: `pending_approval -\u003e\narmed` requires an explicit `Approve` call, and any edit to an armed\nbinding reverts it.\n\n### 3. Rate and volume limits per source\n\nCapture writes are bounded independently of the approval gate, so a flood\nof unauthenticated traffic cannot exhaust disk or drown the inspector:\n`InsertCapture` (`internal/store/captures.go`) prunes by both **retention\nwindow** (`WHERE received_Line truncated
},
{
"url": "/docs/architecture/configuration/",
"title": "Configuration",
"section": "Architecture",
"body": "# Configuration\n\n**Status:** Approved; existing-type assignment is ongoing \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Ownership\n\nConfiguration has two separate responsibilities:\n\n1. `internal/infrastructure/configuration` loads files, environment variables,\n secret references, overlays, and compatibility representations.\n2. Each domain and plugin owns the typed runtime settings required for its\n behaviour.\n\n`internal/app` translates external input into those settings, validates\ncross-capability combinations, and supplies each component only its own\nsettings.\n\nThe following are prohibited:\n\n- passing complete application configuration into a domain;\n- making a domain depend on TOML, YAML, environment-variable names, secret\n references, or a configuration library;\n- defining a central runtime configuration type imported across the system;\n- placing plugin options in an unrelated global configuration model.\n\n## Private input document\n\n`internal/infrastructure/configuration` MAY use a complete decoded document\nwhile discovering files, decoding sections, and applying overlays. It is an\ninput-boundary DTO only.\n\nThe document:\n\n- remains private to the configuration infrastructure;\n- is composed from separately defined feature sections;\n- is immediately translated by application composition;\n- is never returned as the application's runtime model;\n- is never passed through runtime components.\n\nExternal DTOs and domain runtime settings are separate types.\n\nPlugins supply external configuration definitions through a narrow extension\ncontract. The core loader MUST NOT gain a field or recognized filename for every\nplugin.\n\nInvalid configuration input MUST NOT replace active valid settings or terminate\nunaffected capabilities. Configuration changes use the candidate validation,\npromotion, audit, and recovery requirements in\n[safe-change-and-recovery.md](/docs/architecture/safe-change-and-recovery/).\n\n## Dissolution of `internal/config`\n\n`internal/config` has no target location. It MUST be dissolved and MUST NOT\nsurvive as a shared package, configuration domain, or renamed global model.\n\nIts loading behaviour moves to `internal/infrastructure/configuration`. Existing\ntypes and methods are reassigned by behaviour. Repository execution policy,\ndispatch policy, retry policy, ecosystem defaults, tool policy, memory settings,\nindexing settings, provider settings, and channel settings move to their\nrespective owners.\n\n## Repository settings\n\nThe existing `config.Repo` MUST NOT move intact into a repository domain or\nshared settings package. A managed repository is an application association:\n\n- work intake owns which repository references an identity monitors;\n- task execution owns gates, preflight, test protection, ecosystem behaviour,\n and change limits;\n- task lifecycle owns retry behaviour;\n- scheduling owns concurrency policy;\n- workspace management owns base branch and persistent-storage requirements.\n\nApplication composition correlates these settings by repository reference. No\ndomain receives a universal repository profile.\n\n## Documentation\n\nGenerated configuration reference describes external sections and their owning\ncapabilities. It MUST NOT establish a global Go configuration structure as the\npublic contract.\n\n## Value classification rule\n\nEvery value that controls behaviour MUST have an explicit owner and MUST be\nclassified as exactly one of:\n\n1. **Invariant** — changing it changes the meaning of the program, a wire\n protocol, or a domain concept. It may be a Go constant.\n2. **Runtime setting** — an operator, identity, domain, plugin, or deployment\n may legitimately choose a different value. It is typed configuration owned by\n the affected capability.\n3. **Policy value** — it expresses what is allowed, limited, routed, retried,\n protected, or gated. Its typed definition belongs to the owning domain and it\n is evaluated through the policy system.\n4. **Derived value** — it is computed from other settings and is not\n independently mutable.\n5. **Runtime state** — it changes as the application operates, such as the\n active model for a conversation. It is changed by a domain command and is not\n configuration merely because it can be changed.\n\nAn unclassified behavioural literal is a defect.\n\n`const` MUST NOT be used merely because a value currently has one default.\nConstants are reserved for actual invariants: enum discriminants, protocol\nversions, fixed compatibility tokens, and values whose alteration would change\nthe program's semantics rather than configure an installation.\n\nIn Go, a constant is a compile-time value and does not represent mutable runtime\nstorage. It generally has no address that can be observed or changed.\nArchitecturally, a constant therefore asserts that the value cannot chLine truncated
},
{
"url": "/docs/architecture/dependencies-and-contracts/",
"title": "Dependencies and Contracts",
"section": "Architecture",
"body": "# Dependencies and Contracts\n\n**Status:** Approved foundation \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Dependency direction\n\n```text\ndomain and shared application contracts \u003c- infrastructure implementations\n ^\n |\n application composition\n```\n\nThe following rules are strict:\n\n1. Domains MUST NOT import `internal/infrastructure`.\n2. A domain defines the smallest interfaces required to perform its work.\n3. Infrastructure implements those interfaces and translates external\n representations at the boundary.\n4. Only application composition chooses concrete implementations.\n5. A domain MUST NOT manipulate another domain's state or call its internal\n services.\n6. Cross-domain work uses an explicitly designed command, event, or approved\n shared contract.\n7. Implementation difficulty is not permission to violate a boundary.\n\nBroker messages, database rows, SDK response types, and configuration DTOs MUST\nNOT leak into domain models.\n\n## Commands and events\n\n- An event records something that happened and is owned by its publishing\n domain.\n- A command requests an action and is owned by the domain expected to handle it.\n- Consumers do not redefine an event's meaning.\n- Senders obey the receiving domain's command contract.\n- There is no generic package containing unrelated application messages.\n\n## Event bus\n\n`internal/eventbus` is an approved shared application contract. It defines only\nthe messaging behaviour Archie requires, including publication, subscription,\ndelivery outcomes, and required lifecycle semantics.\n\nIt MUST remain broker-neutral. NATS subjects, JetStream configuration, consumer\ngroups, acknowledgements, reconnection, and serialization belong under\n`internal/infrastructure/eventbus/nats` unless Archie explicitly requires a\nsemantic guarantee in the shared contract.\n\nMessage schemas stay with their owning domains. The event bus transports\nmessages; it does not own their meaning.\n\n## Wire contracts and generation\n\nCanonical worker and service wire contracts and their authoritative generation\ninputs are owned by the domain or capability that defines their meaning. They\nMUST NOT be centralized in `internal/infrastructure/spec` or another generic\nschema package.\n\nInfrastructure owns:\n\n- generic contract-generation machinery;\n- transport-specific adapters and generated implementations;\n- RPC, REST, NATS, and other external encoding and delivery concerns.\n\nFor example, agent-worker requests and results belong to task execution.\nSubprocess, NATS, or HTTP representations are infrastructure implementations of\nthat contract. The same rule applies independently to forge, persistence,\nworkspace, and other service contracts.\n"
},
{
"url": "/docs/architecture/generated-documentation/",
"title": "Generated Documentation",
"section": "Architecture",
"body": "# Generated Documentation\n\n**Status:** Generator scaffold in place; renderer and publishing removed\n**Date:** 2026-07-28 (revised 2026-09-12)\n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nArchie's documentation-generation approach is:\n\n- production Go code, registries, configuration, and CLI behaviour remain the\n source of truth for published contracts;\n- a repository-local Go generator extracts and normalizes documentation data;\n- the generator writes deterministic JSON/data files and protocol schemas;\n- the rendering and publishing surface is an **open decision** — the VitePress\n site and its Pages deployment were removed on 2026-09-12 as an unnecessary\n build step (see \"Rendering and publishing\" below).\n\nHugo is not part of the target architecture.\n\nThe documentation system is rooted at:\n\n```text\ntools/\n go.mod\n go.sum\n docsgen/\n main.go\ndocs/\n data/generated/\n public/schemas/\n *.md\n```\n\n### Rendering and publishing — OPEN\n\nSuperseded 2026-09-12. The repository previously carried a VitePress site\n(`docs/.vitepress/config.mts`, `docs/package.json`, a 1620-line\n`docs/pnpm-lock.yaml`, `docs/pnpm-workspace.yaml`) built and deployed to GitHub\nPages by `.github/workflows/docs.yml`. It was removed as an unnecessary build\nstep: the published site was never stood up, and the Pages target is not\nconfigured.\n\n`docs/` is now repository documentation only — markdown read in the repository,\nan editor, or a forge's file view. Nothing in the repository builds, renders, or\npublishes it.\n\nThe renderer for the generated `data/generated/*.json` is therefore **undecided**.\nConsequences that follow, and their current state:\n\n| Concern | State after removal |\n| --- | --- |\n| Reference-page rendering | Undecided; `data/generated/contracts.json` is committed but unrendered |\n| Internal link validation | Not enforced by any build; links are repository-relative (`../architecture/...`) and resolve on the filesystem |\n| Generated-data drift detection | Unchanged — `docsgen check` compares committed output, independent of any renderer |\n| Navigation/sidebar | The authoritative index is `docs/prds/01-project-management.md` |\n| Local authoring server | None; read markdown directly |\n\nA future renderer MUST NOT reintroduce a dead-link-blind build: VitePress\npreviously validated internal links during its production build, and no\nreplacement currently does.\n\n## Single source of truth\n\nEvery published capability has one authoritative machine-readable definition\nowned by its domain or application boundary. Generated documentation derives\nfrom those definitions.\n\nGenerated reference includes:\n\n- domain commands and events;\n- Messages and typed message content;\n- Agent and Workflow contracts;\n- WorkflowSteps and Workflow plugin capability contracts;\n- policies and their typed inputs and outcomes;\n- tools and capabilities;\n- CLI commands, arguments, flags, defaults, and help;\n- runtime settings, types, defaults, validation, and secret metadata;\n- public worker and transport contracts;\n- indexes of registered functionality;\n- deprecation information.\n\nHandwritten documents contain:\n\n- architecture decisions and rationale;\n- tutorials and operational guides;\n- examples and walkthroughs;\n- migration guidance;\n- conceptual explanations.\n\nHandwritten documents link to generated reference rather than copying\nauthoritative tables.\n\n## Target generation architecture\n\n`tools/docsgen` is Archie's documentation extractor and normalizer. It produces\nstructured documentation data and protocol specifications. It does not own\npage layout and MUST NOT generate Markdown pages or fragments.\n\nThe generator is divided internally into reusable stages:\n\n```text\nload registries and types\n -\u003e normalize documentation model\n -\u003e validate ownership and identifiers\n -\u003e serialize data and protocol artifacts\n -\u003e compare or write output\n```\n\nThe normalized documentation model prevents each output renderer from\nindependently interpreting Go types. It contains stable documentation IDs,\nowners, names, descriptions, source references, versions, deprecations, fields,\ndefaults, validation, secret classifications, and relationships.\n\nInitial generator targets:\n\n```text\ndocsgen data\ndocsgen asyncapi\ndocsgen all\ndocsgen check\n```\n\n- `data` writes the normalized JSON reference data.\n- `asyncapi` writes protocol schemas and envelopes.\n- `all` generates every committed artifact.\n- `check` generates into a temporary directory and fails on drift.\n\nThe command may use subcommands or an equivalent explicit `--target` flag. It\nMUST retain one normalization and validation path.\n\n## Target output\n\nDuring adaptation, `docs` remains the staging site. Generated outputs live in\na visibly owned subtree:\n\n```text\ndocs/\n data/\n generated/\n cataloLine truncated
},
{
"url": "/docs/architecture/identity/",
"title": "Identity",
"section": "Architecture",
"body": "# Identity\n\n**Status:** Approved foundation; lifecycle details are deferred \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nIdentity answers which persistent actor owns or performed an action. It supplies\nstable cross-domain attribution and lifecycle semantics.\n\nIdentity kinds include:\n\n- the Archie system identity, such as `System` or `archied`;\n- agent or bot identities;\n- service-account identities;\n- future user identities.\n\nA user is another kind of identity rather than a separate foundation concept.\n\n## Ownership\n\n`internal/domain/identity` owns:\n\n- immutable canonical `IdentityID` values;\n- identity kind;\n- lifecycle state;\n- ownership and attribution semantics;\n- identity-related commands and events.\n\nDomains may record an identity reference on their own state and events when\nownership or attribution is required. They MUST NOT receive a complete\nper-identity application configuration bundle.\n\nPersona and communication style are separate from identity.\n\nEvery Agent has its own immutable acting `IdentityID`. Agent actions use that\nidentity for ownership and attribution in the same way an organisation records\nwhich user accessed a resource, changed a permission, or performed another\nauditable action.\n\nAgent and Identity remain separate aggregates. Identity supplies the stable\nactor reference; Agent owns assistant behaviour, memory, specialisation,\nconversations, and capability context.\n\nAgent identities MUST NOT be shared between Agents. Automated WorkflowExecutions\nMAY eventually act through a dedicated service-account identity when a specific\nunattended automation requires it. Such use is an explicit execution decision\nand audit principal, not the default for Agents or a substitute for recording\nthe initiating user when one exists.\n\nA Workflow is owned by an IdentityID and executes as its owner by default.\nFuture delegated execution MAY explicitly assign a different `RunAsIdentityID`,\nincluding a user or service-account identity. Ownership, effective execution\nidentity, initiating identity, and delegation provenance remain distinct audit\nfacts.\n\n## Canonical identity\n\nEvery identity has an immutable `IdentityID`. It is the only value used for\ncross-domain ownership, attribution, routing, persistence, commands, and events.\n\nIdentity kind is explicit:\n\n- system;\n- bot;\n- service account;\n- user.\n\nDisplay names are mutable and MUST NOT be used as identifiers. Forge usernames,\nchannel bot names, Git authors, provider account names, and other external\nidentifiers are bindings associated with an identity rather than identity\nitself.\n\n`System` is represented by a real identity and `IdentityID`. The empty string\nMUST NOT retain special meaning as a legacy system identity.\n\nConfiguration refers to identities by `IdentityID` and supplies external\nbindings and capability settings separately. Renaming an identity or changing an\nexternal account MUST NOT change ownership or historical attribution.\n\nSecret-bearing channel bindings remain configuration concerns rather than\nidentity data. In particular, `[chat.telegram].token` is a\n`secret.SecretRef` resolved by the application through the configured secret\nregistry; the older `token_env` field remains a compatibility fallback. Neither\nform changes the identity that owns or performs a chat action.\n\n## Persistence\n\nIdentity records are durable domain state. The Identity domain's repository is\nthe source of truth; infrastructure implements its persistence contract.\n\nConfiguration MAY bootstrap a missing identity through an explicit, audited\ncreation operation and MAY propose runtime capability bindings for an existing\n`IdentityID`. Configuration is not the identity registry.\n\nOmitting an identity from configuration stops assembling the omitted runtime\ncapabilities. It MUST NOT delete the identity record, erase history, or make\nhistorical references unresolvable.\n\nDisabling or retiring an identity requires an explicit Identity-domain command.\nRetired identities remain resolvable for historical attribution. A failed\nexternal binding affects capability availability without destroying or silently\nreplacing the identity.\n\n## Lifecycle\n\nIdentity lifecycle is deliberately small:\n\n- `active`: may participate in runtime composition;\n- `suspended`: deliberately prevented from initiating new work while ownership\n and history remain;\n- `retired`: terminal, unable to initiate work, and permanently retained for\n attribution.\n\nAllowed transitions are:\n\n```text\ncreate -\u003e active\nactive -\u003e suspended\nsuspended -\u003e active\nactive -\u003e retired\nsuspended -\u003e retired\n```\n\n`retired` has no outgoing transition.\n\nCapability health is separate from identity lifecycle. Forge, model, channel,\nworkspace, and other bindings report health independently. An active ideLine truncated
},
{
"url": "/docs/architecture/messaging-and-work-intake/",
"title": "Messaging and Work Intake",
"section": "Architecture",
"body": "# Messaging and Work Intake\n\n**Status:** Current state researched; foundation decisions in progress \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nArchie is a personal assistant and general agentic system. Its interactions\ninclude conversation, questions, notes, reminders, goals, message handling,\ntroubleshooting, infrastructure operations, server management, source changes,\nand open-ended future capabilities.\n\nA message to an agent is therefore not inherently a work request, and invoking\nan agent is not inherently a WorkflowExecution.\n\nMessaging and work intake are separate from the Agent System's durable workflow\nexecution model:\n\n- messaging preserves interactions with people and external systems;\n- an agent turn interprets and responds to an interaction using its available\n capabilities;\n- work intake handles the optional promotion of an interaction into durable,\n workflow-backed work;\n- the Agent System owns Workflow definitions and creates WorkflowExecutions.\n\nForge issues, Jira issues, Telegram messages, emails, webhooks, and future\nchannels remain peer external message or artifact sources. None becomes an\nexecution model merely because it can lead to work.\n\n## Confirmed boundary\n\nWork intake MUST NOT create a WorkflowExecution directly.\n\nInbound interactions normally route to the Agent selected by the channel\nbinding. The selected Agent retains its own memory, user knowledge, conversation\ncontinuity, specialisation, and capability context. This continuity is\nindependent of the underlying model or provider.\n\nEach selected Agent acts through its own immutable IdentityID. Interaction and\nresulting action records preserve both that acting identity and the initiating\nuser identity when one exists.\n\nAgent memory is private by default. Interaction context MAY additionally use\nuser-wide memory and intentionally global shared memory, but another Agent's\nagent-wide memory is not part of the selected Agent's context.\n\nMultiple users may interact with the same Agent. The channel binding and inbound\nmessage MUST resolve both AgentID and initiating User IdentityID before\nconversation or memory context is loaded. Sharing an Agent MUST NOT implicitly\nshare one user's private memory with another user.\n\nAn interaction may assemble memory from four scopes: global shared, the selected\nAgent's agent-wide memory, the initiating user's authorized user-wide memory,\nand the private Agent-user relationship memory addressed by both IDs.\n\n## Conversation identity\n\nEvery conversation is identified by the complete tuple:\n\n```text\nAgentID\n+ UserIdentityID\n+ ChannelBindingID\n+ ExternalConversationID\n+ ThreadID\n```\n\n`ThreadID` has an explicit empty value for channels without threads. No element\nof this key may be replaced by a display name, configured bot username, channel\ntype, or other mutable label.\n\nAgent and user memory continue across conversations according to their memory\nscope. Immediate conversation history and delivery correlation remain isolated\nto the identified channel conversation or thread. Moving between Telegram,\nemail, or another channel does not create a new Agent or user, but it does\ncreate a distinct conversation unless an explicit linking operation relates\nthem.\n\nConversation and message events support projections for channel-entry metrics,\nuse cases, volumes, outcomes, latency, capability usage, and other product or\noperational analysis. Metrics are derived read models; they MUST NOT become\nmutable counters embedded in the conversation aggregate or change message\nrouting semantics.\n\n## Canonical messages\n\nEvery inbound, assistant, tool-call, and tool-result message has an immutable\napplication-generated MessageID. Platform identifiers are external correlation,\nnot substitutes for the canonical ID.\n\nMessages are addressable because an interaction is more than text. A message may\ncontain assistant content, reasoning data subject to its retention policy, one\nor more tool calls, a tool result with authoritative execution metadata,\nattachments, reply relationships, edits, or other typed content. Stable\nMessageIDs also allow sessions to branch from an exact prior point.\n\nThe canonical session and message implementation must supply:\n\n- application-generated stable message IDs;\n- user, assistant, and tool roles;\n- assistant tool calls and correlated tool-result messages;\n- authoritative tool-result status, timing, error, truncation, and size\n metadata;\n- session lifecycle and active-request state;\n- session-to-agent-instance and parent-session lineage;\n- durable SQLite round trips and timestamp validation.\n\nThese semantics must use the confirmed Agent, user, channel-binding, and\nconversation ownership model.\n\nThe canonical message store MUST preserve messages as immutable records and\nrepresent later edits, deletions,Line truncated
},
{
"url": "/docs/architecture/migration-decisions/",
"title": "Architecture Migration Decisions",
"section": "Architecture",
"body": "# Architecture Migration Decisions\n\n**Status:** Active migration inventory \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nThis document records what remains to be decided before Archie's current\nimplementation can be migrated into the approved domain-oriented structure.\n\nThe product and core domain model are not being redesigned. Existing Archie\nbehaviour and the existing session implementation are the migration\nbaseline. Remaining decisions MUST be grounded in current packages, consumers,\ntests, persistence, and runtime wiring.\n\n## Confirmed domain packages\n\n```text\ninternal/domain/\n identity/ (approved, not yet materialised as a package)\n agent/ (approved, not yet materialised as a package)\n workflow/\n messaging/ (canonical Message/Conversation types only; gateway migration pending, see section 2)\n workintake/\n plugin/ (approved, not yet materialised as a package)\n curator/\n mapping/\n scheduling/\n```\n\n- `identity` owns persistent actors, lifecycle, ownership, attribution, and\n audit identity.\n- `agent` owns persistent assistants, specialisation, instructions,\n model-independent continuity, capability context, and associations with\n identity and memory.\n- `workflow` is a separate domain. It owns reusable, versioned Workflow\n definitions, WorkflowSteps, WorkflowExecutions, StepExecutions, Attempts,\n outcomes, and typed Workflow plugin contracts.\n- `messaging` owns canonical Messages, Conversations, branches, channel and\n source correlation, replies, acknowledgements, delivery attempts, events, and\n projections.\n- `workintake` owns admission, validation, deterministic routing, and accepted\n work requests for the optional transition into durable workflow-backed\n execution.\n- `plugin` owns generic plugin identity, discovery, compatibility, and lifecycle\n mechanics. It remains metadata-only; capability semantics and typed\n registration belong to the owning domain.\n- `curator` owns the curator engine family: long-running agent loops that\n reason over memory and perform maintenance tasks on a declared shape\n (check-in interval, tool set, attached memory engine). See epic\n archie-core-yp9.\n- `mapping` owns the payload field-mapping vocabulary: binding named fields to\n JSON paths inside a captured webhook payload, and detecting drift. See\n `docs/prds/payload-field-mapping.md`.\n- `scheduling` owns the ticker engine: deciding when a scheduled job runs and\n with what concurrency. It declares `JobSource`/`Runner` contracts and owns\n no storage or delivery. See epic archie-core-1786637496320-249-be6d9a20.\n\n**Reconciliation (2026-08-31):** `curator`, `mapping`, and `scheduling` existed\nin the tree without being added to this list when their epics landed. They are\nconfirmed as legitimate, narrowly-scoped domains under the same rules as the\noriginal six — each owns one capability family, declares typed contracts, and\nhas no cross-domain reach. This list was out of date, not the tree.\n\nConversely, `identity`, `agent`, `messaging`, and `plugin` are approved target\ndomains that do not yet exist as packages. Their behaviour currently lives in\npre-migration packages (see \"Complete package destination map\" below for the\nfull inventory). Do **not** create empty stub packages for them ahead of an\nactual migration PR — an unused stub is dead code and gives no test coverage.\nEach materialises when its corresponding migration decision below is resolved\nand implemented in the same change.\n\nMemory's final package placement remains undecided pending focused review of the\nexisting implementation.\n\nThe following are approved shared or composition packages, not domains:\n\n```text\ninternal/eventbus/\ninternal/policy/\ninternal/infrastructure/\ninternal/app/archied/\ninternal/app/agentworker/\n```\n\n## Remaining migration decisions\n\n### 1. Complete package destination map\n\nEvery current package MUST receive one explicit disposition:\n\n- move into a confirmed domain;\n- move under infrastructure as an implementation of a domain-owned contract;\n- move into application composition;\n- merge with an identified duplicate;\n- remain as an approved shared application contract;\n- move to a plugin, extra, deployment, example, or generated artifact;\n- be deleted after its replacement is proven.\n\nThe largest unreviewed areas include memory, tools and capabilities, workspace\nand worktree behaviour, gates, storage, events, web UI, containers, RPC\npackages, release management, and scheduling.\n\nThe destination matrix MUST name the behaviour owner, target path, dependencies\nto remove, state and contracts to preserve, migration prerequisites, and\ndeletion criteria.\n\n**`internal/webui` / UI Service — RATIFIED Phase 3 disposition (2026-09-07).**\nThe current owner is the in-process HTTP/SSE dashboard: SPA hosting, operator\nauthLine truncated
},
{
"url": "/docs/architecture/observability/",
"title": "Task Observability",
"section": "Architecture",
"body": "# Task Observability\n\n**Status:** Implemented. The per-run detail surface was added on 2026-09-18 and\nis described below.\n**Date:** 2026-08-09 (revised 2026-09-18)\n\n## Historical problem\n\narchied's own daemon logging (`internal/logging/`) was already solid: a\nrotating file sink, a live `Feed` the dashboard subscribes to, and a reader\nthe `/api/logs` endpoint serves from. Before this implementation, none of\nthat reached inside a task.\n\nA task's actual work happens in a task-scoped `archie-agent` container,\ndispatched as one complete workflow over NATS.\nBefore this design was implemented, five things destroyed the evidence of what\nthat container did:\n\n1. **Worker startup** constructed an stderr-only logger and never replaced its\n handler after discovering a dedicated boot task, so runtime records could\n not leave the container.\n2. **`internal/container/pool.go`** creates every task container with\n `AutoRemove: true`. The container — and its stderr — is gone at exit,\n success or failure alike.\n3. **`internal/domain/workflow/implement.go`, `StageBaselineGate`** captures\n `go build`/`go vet`/`go test` output via `cmd.CombinedOutput()`, then\n clips it to 200 characters into a `Warn` log line and discards the rest.\n The eventual park reason is built from `res.Status` alone — the actual\n compiler error is never persisted anywhere a human or Archie's own chat\n tools can read it back.\n4. **`agentexec.Result`** (`internal/agentexec/protocol.go`) carries only\n `Summary`/`Detail` — no step-by-step transcript (tool calls, durations,\n errors). There is no transcript to recover even in principle.\n5. **`agentexec.SubjectForSystem`** is documented as the return channel for\n exactly this — *\"log dumps, health, and PII warnings... the daemon reads\n these for observability and never forwards them\"* — but has no publisher\n and no consumer anywhere in the codebase except a subject-format\n assertion in `nats_test.go`. The designed return path was never wired.\n\nThe practical consequence: a task parks with\n`stage baseline: baseline red -- go build ./... fails and builder could not\nauto-fix (status: parked)` and there is no way — not from the dashboard, not\nfrom Telegram chat, not by SSH'ing into the deployment host — to see *why*\n`go build` failed. The container that knew is already gone.\n\nThis matters beyond debugging convenience: supported deployments can combine\na resident daemon with ephemeral Docker containers that run unattended for\nweeks between operator logins, and the system is\nintended to be usable by people other than the original operator. A system\nthat is only debuggable by an operator who happens to be watching `docker ps`\nin real time does not meet that bar.\n\n## Decision: NATS is the sole autonomous handoff\n\nThe agent image must reach NATS in every deployment, because the broker is the\nonly autonomous task handoff and the only channel carrying task results back\nto the daemon. `archied` has no workflow runner or host fallback. A container\nthat dies before establishing a NATS connection is a distinct, narrower\nfailure (connectivity/config) from the one this document addresses (a connected\ncontainer's own output being unrecoverable), and is out of scope here.\n\n## Implemented design\n\nThe key property: **logs leave the container while it is alive**, so its\ndeath at `AutoRemove` no longer matters. Persistence happens in the daemon\nprocess, which already owns durable state.\n\n### Sink: extend `internal/logging`, not a new package\n\n`internal/logging` already owns its on-disk format end to end per\n`organisation.md` (\"a package owns its own format end to end\"). Add a\ntask-scoped writer producing the same `logging.Entry` JSONL shape at\n`\u003cstate_dir\u003e/logs/tasks/\u003ctask_id\u003e/attempt-\u003cn\u003e.jsonl`. `logging.Tail`,\n`logging.Query`, and `logging.Components` operate on this unchanged;\n`internal/webui/api_logs.go` stays transport-only, as it does today.\n\n### Agent side: make the designed subject real\n\n`internal/app/agentworker` replaces the stderr-only `slog.Handler` with one\nthat tees to stderr (kept for container-level operator debugging when a\nshell is attached) and publishes each record to\n`agentexec.SubjectForSystem(taskID)`, which already exists for this purpose.\nThe application layer obtains only a narrow publisher from\n`internal/infrastructure/agenttransport/nats`; raw core-NATS types remain in\ninfrastructure. `internal/infrastructure/agentboot.TaskID` reads the task ID\nfrom `\u003cworktree\u003e/.git/task.json` per the container/pool.go trap documented in\n`CLAUDE.md`. Publishing is fire-and-forget: log shipping must never block or\nfail the run it is reporting on.\n\n### Daemon side\n\n`cmd/archied/main.go` registers `subscribeSystemLogs` once at startup on\n`agentexec.SubjectSystemWildcard`. It demultiplexes by task ID, appends to that task's\nsink, and pLine truncated
},
{
"url": "/docs/architecture/organisation/",
"title": "Repository Organisation",
"section": "Architecture",
"body": "# Repository Organisation\n\n**Status:** Approved \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Target structure\n\nArchie MUST be organised around cohesive application responsibilities:\n\n```tree\ncmd/\n archied/\n archie-agent/\n\ninternal/\n domain/\n \u003carea\u003e/\n eventbus/\n policy/\n infrastructure/\n configuration/\n eventbus/\n nats/\n forge/\n persistence/\n \u003cexternal-service\u003e/\n app/\n archied/\n agentworker/\n\nplugins/\n \u003cplugin\u003e/\ndeployments/\n \u003csupported-assembly\u003e/\ndocs/\n guides/\nexamples/\n \u003ccopyable-example\u003e/\nextras/\n \u003coptional-extension\u003e/\n```\n\nDomain names are chosen from application behaviour during package review.\nExisting names MUST NOT be retained merely to reduce migration work.\n\n## Domain areas\n\nA domain owns one cohesive job performed by Archie. It contains that job's\nlanguage, state, rules, operations, commands, events, policy implementations,\nruntime settings, and required service contracts.\n\nCode that changes together SHOULD live together. Empty or ceremonial `entities`,\n`repositories`, or `services` layers are prohibited.\n\n## Application composition\n\n`internal/app`:\n\n- constructs domains and infrastructure implementations;\n- translates external configuration into domain and plugin settings;\n- connects commands, events, and shared contracts;\n- starts services in dependency order;\n- owns health aggregation and shutdown ordering.\n\n`cmd/*` parses only process-level input and invokes the relevant application\ncomposition package. It MUST NOT contain substantive wiring or act as a service\nlocator.\n\n## Process boundaries\n\nArchie begins as a modular monolith even when capabilities run in separate\nprocesses. A domain becomes a separately deployed service only for a concrete\nneed such as security isolation, independent scaling, failure containment, or\nindependent operation.\n\nPossible future extraction does not justify premature network protocols or\nduplicated service scaffolding.\n\nA new command is not a reason for a new binary. `cmd/` entries are process\nboundaries: a binary earns its place when its surface fails independently of the\nfleet, or when one process may need to run in more than one location, such as a\nhigh-availability pair or across a NAT boundary. Operational commands over a\nsurface an existing binary already owns are subcommands of that binary. Offline\ndatabase maintenance belongs to `archie-state-store`, which owns the file and\nits path layout, not to a new recovery binary.\n\n## Optional and operational material\n\n- `extras/\u003cextension\u003e` contains a maintained optional extension with real\n behaviour. Core Archie MUST NOT import it, and removing `extras/` MUST NOT\n prevent the core application from building or running.\n- `deployments/\u003cassembly\u003e` contains a supported assembly, including deployment\n definitions and concise operational instructions.\n- `docs/guides/` contains handwritten explanation and links to generated\n reference material.\n- `examples/` contains copyable samples with explicit purpose, prerequisites,\n and support expectations.\n\nSupported assemblies SHOULD demonstrate that capabilities can be selected\nindependently.\n"
},
{
"url": "/docs/architecture/package-review/",
"title": "Domain and Feature Review Procedure",
"section": "Architecture",
"body": "# Domain and Feature Review Procedure\n\n**Status:** Active \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nArchitecture work proceeds one domain or feature at a time. Before any design\nquestion, the selected area is researched across its complete live\nimplementation, consumers, tests, state, configuration, and runtime wiring.\n\nExisting packages are evidence, not the unit that drives the interview.\nPlacement follows behaviour and ownership, not the existing directory name.\n\nNo restructuring implementation begins until the relevant package decisions\nare agreed and recorded in the focused domain or requirement document.\n\n## Required research\n\nBefore design questions, the current-state report MUST identify:\n\n- entry points and runtime composition;\n- state, persistence, and lifecycle;\n- public and internal contracts;\n- commands, events, and policy enforcement;\n- configuration and defaults;\n- all consumers and cross-domain interactions;\n- process and infrastructure boundaries;\n- tests, failure behaviour, and known invariants;\n- duplicated or conflicting implementations;\n- documentation that disagrees with live code.\n\nDiscoverable facts MUST be investigated rather than presented to the user as\ndesign questions.\n\n## Required decisions\n\nEach package review records:\n\n| Decision | Required result |\n|---|---|\n| Owned behaviour | The cohesive job the code performs |\n| Domain ownership | The owner of the behaviour and vocabulary |\n| Final location | Its target path |\n| Public contracts | The smallest interfaces, commands, and events exposed |\n| Required services | Domain-defined interfaces implemented by infrastructure |\n| State ownership | The owner of each mutable record and transition |\n| Dependencies | Allowed imports and dependencies to remove |\n| Duplication | Code to consolidate, delete, or keep separate |\n| Plugin status | Core domain, shared contract, infrastructure, plugin, or extra |\n| Process boundary | In-process, or the concrete reason for isolation |\n| Configuration | Settings owned by the domain or plugin |\n| Policy | Definitions, evaluators, consequences, and shared mechanics used |\n| Documentation | Authoritative definitions and generated reference |\n| Migration constraints | Behaviour and data that must remain correct |\n\n## Interview method\n\nFor every selected domain or feature:\n\n1. Declare the single area under review.\n2. Complete the current-state research.\n3. Explain actual behaviour in plain technical language.\n4. Separate facts from design decisions and concerns.\n5. Resolve obvious consequences directly and ask only questions whose answer\n materially changes behaviour or structure.\n6. Record each confirmed decision in its focused document.\n7. Complete that area's handoff before selecting another.\n8. Keep unrelated domains outside the implementation scope.\n"
},
{
"url": "/docs/architecture/plugins-and-extensions/",
"title": "Plugins and Extensions",
"section": "Architecture",
"body": "# Plugins and Extensions\n\n**Status:** Approved foundation \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Plugin framework\n\nThe core plugin domain lives under `internal/domain/plugin`. It owns:\n\n- plugin identity and metadata;\n- discovery and registration contracts;\n- lifecycle rules;\n- capability registration;\n- validation and compatibility rules.\n\nThe generic plugin contract remains metadata-only. Functionality is exposed\nthrough narrow typed capability contracts with an owning registry or manager.\n\nWorkflow plugins belong to the Workflow capability family. The Workflow domain\nat `internal/domain/workflow` owns their typed registration, validation,\nversioning, definition, and execution contracts. The generic plugin domain may\nprovide plugin identity, discovery, compatibility, and lifecycle mechanics, but\nit MUST NOT define Workflow semantics or expose untyped Workflow hooks.\n\n## Plugin implementations\n\nFirst-party plugins are self-contained vertical modules under\n`plugins/\u003cplugin\u003e`.\n\nA plugin:\n\n- owns its implementation, settings, templates, assets, and tests;\n- receives only explicitly supplied contracts and services;\n- MUST NOT receive daemon internals, a service locator, or unrestricted\n application access;\n- MUST NOT import arbitrary infrastructure or application composition code;\n- MAY use approved domain and shared application contracts;\n- MUST remain removable without changes to unrelated domains.\n\nPlugins may be internally cohesive like small services. They do not become\nnetwork services without a concrete security or operational reason.\n\n## Event sources and reactions\n\n`archie-core-7d5u.1` asked whether extensions should react to events\nin-process through typed capability families (this document's model) or\nout-of-process like a generic `DispatchEvent` dispatcher. Decided 2026-08-22\nin `docs/prds/event-sources-and-reactions.md`: in-process typed families,\nconfirming this document's model rather than amending it. The forge webhook\nreceiver (`internal/forge/webhook`, see `ARCHITECTURE.md`'s \"Webhook intake\"\nsection) and the EDA playbook engine's Module action position\n(`docs/prds/eda-playbook-engine.md`, `internal/domain/eda/module`) are the\nshipped instances of that decision -- each is a narrow typed contract with an\nowning registry, not a generic event hook. The playbook engine's Channel and\nForge action positions are designed but not yet implemented (open\ninvestigation `archie-core-t2db.19`); when they land they follow the same\nmodel, not a new one.\n\n## Curator surprisal sampling\n\n**Decided 2026-08-05, wave 1 landed 2026-09-05** — the curator engine family\n(epic [#435](https://github.com/samcharles93/archie-core/issues/435)) selects which memories deserve agentic attention via\nsurprisal-based sampling: score memories by how surprising they are and\nspend the sampled reasoner's agentic budget on the most surprising items.\n\nThe prerequisites are built: the embeddings capability\n([#436](https://github.com/samcharles93/archie-core/issues/436), `internal/domain/embedding` contract +\n`internal/infrastructure/embedding` implementation, config-driven via\n`models.embedding`) and the wave-1 `Sampler` strategy seam\n(`internal/domain/sampling` -- not `internal/curator`, see\n`docs/prds/curator-sampler-wave1.md` for the settled design and tracking)\nwith four\ncheap, embedding-free strategies (recency, random, all, staleness proxy).\nNo curator consumes a `Sampler` yet, and the embeddings client is likewise\ndeliberately inert: `setupEmbeddings` constructs it at boot and holds it on\n`b.embeddings`, but no production code reads that field yet, so a resolved\n`models.embedding` role currently produces a client whose only observable\neffect is a boot log line. Both are prerequisites waiting on #437, not\nunwired defects.\n\nStill deferred: [#437](https://github.com/samcharles93/archie-core/issues/437), the embedding-backed surprisal strategy, which\ndepends on the above and on [#407](https://github.com/samcharles93/archie-core/issues/407). Requirements for that deferred work,\nso it can be picked up without re-deriving the design:\n\n- A `Sampler` implementation behind the same interface as the cheap\n strategies; selection deterministic given fixed inputs.\n- Algorithm: sample up to a candidate cap; build a k-NN graph over embedding\n vectors; row-normalize the adjacency with self-loops; power-iterate to the\n stationary distribution; surprisal = −log(stationary probability of the\n nearest point); select the highest-surprisal items.\n- Degradation: content without an embedding is skipped; an embedding failure\n aborts sampling and falls back to free exploration — a sampling failure\n must never fail or crash a curator pass.\n- Cost bound: the graph build is quadratic in the candidate cap; the cap must\n be a named constant with a documentLine truncated
},
{
"url": "/docs/architecture/policy/",
"title": "Policy System",
"section": "Architecture",
"body": "# Policy System\n\n**Status:** Approved foundation; exact API is deferred \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nRetry limits, authorization, gates, commands, protected paths, tool\navailability, and similar rules define or constrain what Archie may do. They\nneed consistent consumption, composition, evidence, and audit without erasing\ntheir different domain meanings.\n\n## Ownership\n\n`internal/policy` is an explicitly approved shared application contract. It is\nnot infrastructure and it is not a generic domain that owns every rule.\n\nThe shared policy system owns:\n\n- policy identity, version, scope, and applicability;\n- typed evaluation invocation;\n- deterministic composition and precedence;\n- common evaluation states such as satisfied, violated, indeterminate, and not\n applicable;\n- reasons and evidence;\n- evaluation audit records;\n- registration, discovery, and lifecycle.\n\nEach consuming domain owns:\n\n- its policy vocabulary and typed inputs;\n- its concrete policy definitions;\n- its evaluators;\n- the meaning and consequence of an evaluation.\n\nAccess therefore owns principals, resources, actions, and authorization\nconsequences. Task lifecycle owns retry eligibility and exhaustion. Task\nexecution owns gate requirements. Workspace mutation owns protected-path\neffects. Tool execution owns tool and command constraints.\n\nThese policies use consistent evaluation mechanics without pretending their\ndecisions are semantically interchangeable.\n\n## Infrastructure\n\nInfrastructure MAY load, persist, distribute, and audit external policy\nrepresentations. It MUST NOT define what Archie allows or assign the meaning of\na domain policy result.\n\nThe exact Go API, composition operators, precedence rules, and representation\nformat require a focused policy design pass.\n"
},
{
"url": "/docs/architecture/safe-change-and-recovery/",
"title": "Safe Change and Recovery",
"section": "Architecture",
"body": "# Safe Change and Recovery\n\n**Status:** Requirements approved; mechanics are under design. Database-backed settings are implemented through the control plane (2026-09): see the degrade/recovery section below, which records a gap rather than a solution. The full runtime-supervision protocol (bounded observation, versioned audit trail) remains under design.\n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nArchie is built and improved by agents as well as humans. The architecture MUST\nmake a change easy to locate, understand, test, audit, extend, and reverse\nwithout requiring its author to understand unrelated parts of the application.\n\nSeparation of concerns is therefore an operational safety boundary, not only a\ncode-organisation preference.\n\n## Required outcomes\n\n- An agent can improve a focused feature without receiving authority over\n unrelated domains.\n- Every event, function, action, setting, and policy has an identifiable owner\n and provenance.\n- Source and runtime changes pass deterministic checks outside the actor making\n the change.\n- A rejected or unhealthy candidate cannot replace the last known working\n version.\n- Invalid user configuration does not terminate an otherwise healthy\n application.\n- Configuration and service failures are observable, attributed, and eligible\n for bounded remediation or rollback.\n- The application continues serving unaffected capabilities when one\n capability is misconfigured or unhealthy.\n- Recovery failure is reported and contained rather than hidden in an\n unbounded retry loop.\n\n## Source changes\n\nAgent-authored source changes occur through the task-execution boundary in an\nisolated workspace. The agent does not directly mutate the running\ninstallation.\n\nPromotion is controlled by deterministic repository-owned policy, including\ntests, generated-reference drift checks, protected paths, change scope, and\nother required gates. The actor proposing a change cannot waive these checks.\n\nRepository and domain separation SHOULD allow a task to load and modify only\nthe governing requirements and code for its declared scope.\n\n## Runtime changes\n\nConfiguration and other runtime changes are proposed as candidates rather than\nmutating the active state in place.\n\nThe affected domain owns:\n\n- the command describing the semantic change;\n- validation of its typed settings;\n- health signals that demonstrate correct operation;\n- domain-specific remediation and rollback consequences.\n\nShared policy supplies consistent evaluation, evidence, and audit mechanics.\nIdentity supplies actor attribution. Infrastructure supplies external\nconfiguration storage, service control, and other operational adapters.\n\n## Runtime change protocol\n\nLast-known-good preservation and candidate promotion are the universal runtime\nchange protocol.\n\n1. An identity proposes an immutable candidate version.\n2. Infrastructure parses it without changing active state.\n3. Each affected domain validates its typed settings.\n4. Shared policy evaluates cross-cutting constraints.\n5. Each affected capability prepares the candidate without destroying its\n active instance.\n6. Capability-specific health checks prove the candidate works.\n7. Runtime supervision atomically promotes the candidate.\n8. A bounded observation period watches for delayed failure.\n9. Failure restores the last-known-good version and records the reason,\n evidence, actor, and involved versions.\n10. Repeated remediation failure stops retrying, isolates the capability, and\n reports the fault.\n\nThe affected domain owns the semantic change command, typed validation, health\nsignals, and remediation consequences. Shared runtime supervision owns the\npromotion transaction. Infrastructure performs storage and service-control\noperations.\n\nThe protocol's exact Go contracts, storage model, atomicity boundary, and\nobservation rules require a focused runtime-supervision design pass.\n\n## Failure isolation\n\nInvalid input MUST leave the active valid configuration unchanged and produce a\ndiagnostic record.\n\nA capability that cannot start or reload SHOULD fail independently where its\nabsence does not violate a declared application invariant. Unaffected\ncapabilities continue operating.\n\nRemediation MUST be bounded, observable, and policy-driven. Recovery MUST NOT\nsilently oscillate between versions or retry forever.\n\n## Database settings: degrade paths and recovery (2026-09)\n\nThe control plane owns runtime-tunable settings and stores them in `archie.db`.\nThis replaced a dashboard overlay store that degraded to file config on any\nfailure and had three documented ways back. The replacement has none, and the\nposture is the opposite: **the daemon fails closed.**\n\n### Degrade paths\n\n| Failure | Boot behaviour | Where the operator sees it |\n| --- | --- | --- |\n| State Store uLine truncated
},
{
"url": "/docs/guides/first-playbook/",
"title": "Building your first playbook binding",
"section": "Guides",
"body": "# Building your first playbook binding\n\nThis walks through the whole no-code path the event-capture epic promises:\npoint an external app's webhook at archie, see the real payload it sends,\nmap the fields you care about, bind them to a workflow, and watch the next\nmatching event turn into a task -- no code, no redeploy, no restart. See\n`docs/architecture/bindings.md` for the model and threat model behind each\nstep below.\n\n## 1. Send a test event\n\nPoint whatever external app you want archie to react to at:\n\n```\nPOST http://\u003cyour-archied-host\u003e:8484/webhooks/capture/\u003csource\u003e\n```\n\n`\u003csource\u003e` is a name you choose for this sender (e.g. `sentry`, `github-ci`,\n`stripe`) -- it's just a URL path segment, not a preset list. Send one real\nevent however is easiest for a first test, e.g.:\n\n```bash\ncurl -X POST http://localhost:8484/webhooks/capture/sentry \\\n -H \"Content-Type: application/json\" \\\n -d '{\"issue\": {\"title\": \"NullPointerException in checkout\"}, \"level\": \"error\"}'\n```\n\nThis event isn't signed yet, so it's captured but marked **unauthenticated**\n-- visible for you to design against, but incapable of triggering anything\n(`docs/architecture/bindings.md`'s threat-model section explains why: an\nunauthenticated event can never match a binding, no matter how it looks).\n\n## 2. Inspect the captured payload\n\nOpen the dashboard's **Event inspector** (`/captures`). Your test event\nappears there with its real body -- this is schema-by-example: you're\nlooking at an actual payload from the actual sender, not guessing at a\nschema in advance.\n\n## 3. Map the fields you care about\n\nOpen **Field mappings** (`/mappings`) → **New mapping**. Pick your captured\nevent from the dropdown, then click through its payload tree on the right --\nevery key and value is clickable. Clicking a value binds it as a field\n(name, JSON path, and an inferred type you can change). Add whichever fields\nyour workflow needs (e.g. the issue title), mark the ones that must be\npresent as **Required**, hit **Preview** to confirm they resolve correctly\nagainst the real payload, then **Save**.\n\n## 4. Create the binding\n\nOpen **Playbook bindings** (`/bindings`) → **New binding**:\n\n- **Name** -- anything descriptive.\n- **Source** -- the same `\u003csource\u003e` segment from step 1 (e.g. `sentry`).\n- **Field mapping** -- the one you just saved.\n- **Workflow** -- which registered workflow should run on a match.\n- **Repo pin** (optional) -- leave both blank to use the single configured\n repo; set owner/repo explicitly if you run a multi-repo deployment.\n- **Shared secret** -- a real secret (16+ bytes), never `test1234`. The\n sender signs every request body with this via HMAC-SHA256 so archie can\n tell a genuine event from anyone who finds the URL.\n\nSave. The binding starts in **draft** -- nothing fires yet.\n\n## 5. Approve it\n\nA **draft** or freshly-**edited** binding needs an explicit human approval\nbefore it can ever match a live event -- this is deliberate, not a missing\nstep: an operator authoring a binding is a considered act, an event arriving\nis not, and nothing should self-arm (`docs/architecture/bindings.md`'s\nthreat-model section, point 2). Click **Approve**. Status becomes **armed**.\n\n## 6. Fire it for real\n\nSend the same kind of event again, this time **signed** with the binding's\nsecret:\n\n```bash\nBODY='{\"issue\": {\"title\": \"NullPointerException in checkout\"}, \"level\": \"error\"}'\nSECRET=\"the-secret-you-set-in-step-4\"\nSIG=$(printf '%s' \"$BODY\" | openssl dgst -sha256 -hmac \"$SECRET\" | sed 's/^.* //')\n\ncurl -X POST http://localhost:8484/webhooks/capture/sentry \\\n -H \"Content-Type: application/json\" \\\n -H \"X-Hub-Signature-256: sha256=$SIG\" \\\n -d \"$BODY\"\n```\n\nThis event is captured as **authenticated**, matched against your armed\nbinding, and dispatched into a new task through the exact same\ngate/worktree/sandbox pipeline every other task goes through -- see\n`ARCHITECTURE.md`'s Task Lifecycle section, \"Provenance by origin.\" Check\nthe **Tasks** page; a new task should appear, its body built from the fields\nyour mapping resolved.\n\n## Editing a live binding\n\nAny edit to an armed binding -- even a typo fix -- drops it back to\n**pending_approval** automatically. This can't be bypassed by editing; it\nhas to be re-approved. When editing, leave the **Shared secret** field blank\nto keep the existing secret; only fill it in if you're deliberately\nrotating it.\n"
},
{
"url": "/docs/prds/config-schema/",
"title": "Backend-owned configuration schema for the webui -- decision",
"section": "PRDs",
"body": "# Backend-owned configuration schema for the webui -- decision\n\n**Status:** Approved\n**Date:** 2026-09-02\n**Beads issue:** `archie-core-b6ew` (epic), with children `.1`-`.5`\n\n## Decision\n\nThe dashboard's Configuration page keeps reading a secret-free,\nhand-picked view of `config.Config` -- that allowlist is a safety\nboundary and stays exactly as it is. What changes is who defines the\npage around it: `ui/src/settings/settings.js` and\n`config-row.js` hardcode every row's label, section, type, and\neditability by hand; after this, the backend attaches that metadata to\neach field it already allows through, and the frontend renders\nsections and fields generically from what the backend sends.\n\nThis is **not** a reflection-based schema over `config.Config`.\n`internal/config/config.go`'s own package doc states it is \"scheduled\nfor dissolution: its types and methods are to be reassigned to the\ndomains whose behaviour they describe\" -- building a generic\nschema-from-struct-tags mechanism over a struct on its way out would be\nbuilding on a foundation this repo has already decided to remove.\nInstead, every field already in `ConfigView` (`internal/webui/api_config.go`)\ngot there because someone deliberately vetted it as safe to show; the\nschema adds a hand-authored descriptor to that same deliberate act,\nnot a new mechanism that re-derives safety automatically.\n\n## Why now\n\nTask #6's investigation (`/9/26`) found the concrete failure this\nproduces: `Repo.AllowConcurrent` and `Repo.MaxRetries` already reach\n`GET /api/config` (`RepoView` in `api_config.go`), but the\nrepositories card renders a static read-only table that never shows\nthem, so the fix for \"why isn't a second agent running\" required an\nagent tracing Go source instead of a checkbox on the dashboard.\n`Repo.ReviewEnabled` is worse: it never reaches the API at all. Both\nare symptoms of the same root cause -- the frontend decides what a\nfield means and whether it's shown, and nothing keeps that in sync\nwith what the backend actually allows.\n\n## Shape\n\n```go\n// internal/webui/config_schema.go (new)\n\ntype ConfigFieldType string\n\nconst (\n FieldString ConfigFieldType = \"string\"\n FieldInt ConfigFieldType = \"int\"\n FieldBool ConfigFieldType = \"bool\"\n FieldDuration ConfigFieldType = \"duration\"\n FieldEnum ConfigFieldType = \"enum\"\n FieldStructured ConfigFieldType = \"structured\" // repositories, models, providers\n)\n\ntype ConfigField struct {\n Key string `json:\"key\"` // dotted path, matches UpdateConfig's key space\n Label string `json:\"label\"`\n Description string `json:\"description,omitempty\"`\n Type ConfigFieldType `json:\"type\"`\n Value any `json:\"value\"`\n Editable bool `json:\"editable\"`\n LockedReason string `json:\"locked_reason,omitempty\"`\n Overridden bool `json:\"overridden\"`\n Options []string `json:\"options,omitempty\"` // enum only\n RestartRequired bool `json:\"restart_required\"`\n}\n\ntype ConfigSection struct {\n ID string `json:\"id\"`\n Label string `json:\"label\"`\n Fields []ConfigField `json:\"fields\"`\n}\n```\n\n`handleConfig` builds `[]ConfigSection` from the existing per-field\nvalues it already assembles, plus `overlay.DeniedKeys` for\n`LockedReason` and the existing `Overridden` list -- both already\ncomputed just re-attached per field instead of returned as\nside-channel maps the frontend cross-references by key string.\n`Locked`/`Overridden`/`Reload`/`Provenance` stay as top-level\n`ConfigView` fields; they describe runtime state (what happened on\nthe last reload, which keys the overlay refuses), not the field\ncatalog, and duplicating that logic per-field would be the same drift\nrisk this change exists to remove.\n\n`FieldStructured` fields (repositories, models, providers) carry\n`Value` as structured JSON and are rendered by dedicated\neditors, not the generic renderer -- see \"What this does not do\".\n\n## Sections\n\nMatches the cards the settings page already has: `identity`,\n`repositories`, `models`, `providers`, `budgets`, `storage`,\n`containers`, `web`. No new sections; this is metadata on existing\nrows, not new configuration surface.\n\n## Frontend\n\n`config-row.js`'s `row(label, value, opts)` becomes the renderer's\nprivate implementation detail, called generically:\n\n```js\nfor (const section of schema.sections) {\n const rows = section.fields\n .filter(f =\u003e f.type !== \"structured\")\n .map(f =\u003e row(f.label, f.value, {\n key: f.key, type: f.type, options: f.options,\n locked: f.locked_reason, overridden: f.overridden,\n restartRequired: f.restart_required,\n }));\n // structured fields render through their dedicated editor (b6ew.4)\n}Line truncated
},
{
"url": "/docs/prds/memory-engine-unification/",
"title": "Memory engine unification: four scopes, CRUD, and one write path -- decision",
"section": "PRDs",
"body": "# Memory engine unification: four scopes, CRUD, and one write path -- decision\n\n**Status:** Approved\n**Date:** 2026-09-16\n**Findings:** `memory-4` and its dependants in the adversarial sweep\n(`.local/issue-tracker`); the real defect is the split described below, for\nwhich `memory-4` is a symptom\n**Supersedes:** the draft `memory-turn-context.md` (its synchronous-read\nargument survives and is reused; its `Manager` method and consumer-owned port do\nnot)\n**Settles:** four open items in `docs/architecture/migration-decisions.md#5`\n(listed at the end)\n\n## Problem\n\nThere are **two live memory stores for one assistant**, split by which component\nwrites them, and the chat turn reads neither.\n\n| Family | Written by | Root | Format |\n|---|---|---|---|\n| `internal/memory` (`Manager`, `builtin.Provider`) | the chat agent's `memory_edit` tool via `memorytoolprovider` (`bootstrap.go`) | `\u003cworkDir\u003e/memory` | `MEMORY.md` / `USER.md` |\n| `internal/domain/memory` + `internal/infrastructure/memory` | the session curator (`sessioncurator.go`) | `\u003cworkDir\u003e/memory-engine` | identity-keyed blocks |\n\nThey share no code, no format and no scope vocabulary. Neither reaches a prompt:\n`Manager.SystemPromptBlock` (`internal/memory/manager.go`) has no production\ncaller anywhere, so the agent can write memory through a tool and never read it\nback.\n\nThree further defects fall out of the same split:\n\n1. **The engine's contract is not CRUD.** `Store` is `Write`/`Query`/`List`/\n `Forget` — create, read, delete, and **no Update**. The retired file provider\n *does* have `handleAdd`/`handleReplace`/`handleRemove`, so the family being\n deleted is operationally richer than its replacement.\n2. **There is no scope model.** The contract carries one opaque `Identity\n string`, and the curator fills it with a **session id** (`sessioncurator.go`)\n — which is none of the four scopes `docs/architecture/agent-system.md`\n requires, and stops meaning anything when the session ends.\n3. **The chat turn has no user identity on half the channels.** Telegram and\n email populate `SenderID` with a person; the dashboard populates it with\n nothing, and a webhook populates it with nothing either — a route path names\n a source, not a person, so `webhook.go` leaves `SenderID` empty and carries\n the route in `Inbound.BudgetKey` for rate limiting. `sender_id` is persisted\n (`session_store_sqlite.go`), so whatever a channel puts there reaches the\n consumers that read it.\n\n`agent-system.md` says the write-scope selection and cross-scope sharing\nrules \"require a focused memory design decision\". This is that decision.\n\n## Decision\n\n### 1. One engine, addressed by four typed scopes\n\n`internal/domain/memory` is the only memory engine. `Observation.Identity`,\n`Query.Identity`, `Record.Identity` and `List(identity)` are deleted, not\nextended: a field whose meaning depends on the caller cannot carry a scope.\n\n```go\ntype ScopeKind string\n\nconst (\n ScopeGlobal ScopeKind = \"global\" // shared across the application\n ScopeAgent ScopeKind = \"agent\" // private to one Agent\n ScopeUser ScopeKind = \"user\" // one user, across their Agents\n ScopeAgentUser ScopeKind = \"agent-user\" // one Agent's relationship with one user\n)\n\ntype Scope struct {\n Kind ScopeKind\n Agent AgentID // required for ScopeAgent, ScopeAgentUser\n User IdentityID // required for ScopeUser, ScopeAgentUser\n}\n\nfunc (s Scope) Validate() error // rejects a component present where it is not required\nfunc (s Scope) Key() string // canonical, collision-free storage key\n```\n\n`AgentID` and `IdentityID` are opaque. This package never resolves or\ncanonicalises them.\n\nA turn's resolved identity is one value:\n\n```go\ntype Subject struct {\n AgentID AgentID\n UserID IdentityID // \"\" when the channel carries no per-person identity\n}\n\nfunc (s Subject) Scopes() []Scope // readable, most specific first\nfunc (s Subject) WritableScopes() []Scope // never includes ScopeGlobal\n```\n\n**Access control is naming a scope, and nothing else.** The read path names\nexactly the scopes of the resolved `Subject`; the engine never unions beyond what\nit is handed and applies no policy of its own. An engine that enforced policy\nwould need to know about channels, sessions and bindings, and every backend would\nreimplement it. \"Agent-wide is private to its owning Agent by default\"\n(`agent-system.md`) falls out mechanically, because `ScopeAgent{a}` and\n`ScopeAgent{b}` are different storage keys.\n\n`ScopeGlobal` is never model-writable. Writing it is an operator action; nothing\nproduces one yet, but the contract models it because the read set includes it.\n\n### 2. CRUD, with revisions retained\n\n```go\ntype Store interface {\n Create(ctx context.Context, in NewRecord) (Record, error)\n Get(ctx context.Context,Line truncated
},
{
"url": "/docs/prds/messaging-service-boundary/",
"title": "Messaging Service boundary and transport design",
"section": "PRDs",
"body": "# Messaging Service boundary and transport design\n\n**Status:** Approved\n**Revision:** 1 \n**Date:** 2026-09-19 \n**Beads issue:** `archie-core-8cda.6.1` \n**Parent:** [`service-decomposition.md`](/docs/prds/service-decomposition/) \n**Prerequisites:** Gateway contract extraction (Phase 1), State Store extraction (Phase 2), UI extraction (Phase 3)\n\nThis document is the authoritative specification for Phase 4: Messaging Service extraction. It fixes the process boundary, configuration ownership, client contracts, security posture, readiness behavior, channel invariants, and deletion criteria required by `archie-core-8cda.6` and its child tasks.\n\n## Decision\n\nThe Messaging Service (`cmd/archie-messaging`, composed via `internal/app/archiemessaging`) owns external channel connections (Telegram, email, webhook) and dispatches inbound messages and commands to the standalone Gateway Service over gRPC via `messaging.ChatContract`.\n\nIt is a peer service to the UI Service and the daemon:\n- It serves external chat participants across configured communication platforms.\n- It consumes Gateway's `messaging.ChatContract` exclusively for chat turn execution, streaming, session queries, cancellation, persona selection, and task control actions.\n- It holds no direct database connections (`*store.Store` or SQLite), owns no LLM execution or model provider credentials, and has no dependency on the daemon runtime or the UI HTTP server.\n\n## Boundary and Ownership\n\n| Concern | Messaging Service owns | Messaging Service consumes | Messaging Service must not own or link |\n|---|---|---|---|\n| **Channel transports** | Telegram bot long-polling, Email IMAP/SMTP loops, Inbound Webhook HTTP listener | Gateway `ChatContract` | Direct LLM providers, model runtimes, tokenizer libraries |\n| **Turn dispatch** | Inbound message packaging, typing indicators, live streaming updates | `ChatContract.Route`, `ChatContract.Stream` | `gateway.Router`, `runtime.Runtime`, agent prompt builders |\n| **Sessions \u0026 History** | Platform-to-session key mapping, conversation thread ID translation | `ChatContract.GetSession`, `RecentMessages`, `RecentTurns` | `*store.Store`, `SessionStore` SQLite handles, direct database files |\n| **Control \u0026 Actions** | Platform command parsing (`/status`, `/tasks`, `/cancel`, `/stop`, `/persona`) | `ChatContract.Snapshot`, `ApplyTaskAction`, `Cancel`, `SetPersona` | `daemon.Daemon`, workflow execution engine, worktree operations |\n| **Task Approvals** | Channel interactive keyboards, approval callbacks | `ChatContract.ApplyTaskAction` | Direct task status mutations, forge comment posters |\n| **Configuration** | Platform bot tokens, allowed sender/user IDs, webhook listen ports | `[chat.*]` channel configs, Gateway target/token | `config.Holder`, forge tokens, repo configs, agent container specs |\n| **Readiness** | Messaging process liveness, channel connectivity, Gateway probe | Gateway `ChatContract.Snapshot` | Internal daemon subsystem probes |\n\n## Target Service Shape\n\n1. **Binary entry point**: `cmd/archie-messaging/main.go` parses CLI flags (`-config`, `-config-overlay`, `-gateway-target`, `-gateway-token`) and hands execution to `internal/app/archiemessaging.Run`.\n2. **Application composition**: `internal/app/archiemessaging/` wires:\n - Config loading scoped strictly to `[chat.*]` and `[services.gateway]`.\n - Secret resolution for channel credentials (e.g. Telegram bot token).\n - Gateway gRPC client (`internal/infrastructure/gatewayrpc.Client`).\n - Channel lifecycle managers (`internal/channels/status`).\n - Channel instances (`internal/channels/telegram`, `email`, `webhook`).\n3. **Turn and Stream Execution**:\n - Channels adapt inbound platform messages to `messaging.Inbound`.\n - Streaming channels (Telegram) consume `ChatContract.Stream(ctx, inbound)` which returns `\u003c-chan messaging.ChatEvent`, mapping `started`, `delta`, `tool`, `media`, `done`, and `error` events directly to live message frames.\n - Non-streaming channels (email, synchronous webhooks) consume `ChatContract.Route(ctx, inbound)`.\n\n## Load-Bearing Channel Invariants\n\nThe Messaging Service preserves existing invariants defined in `AGENTS.md` and channel documentation:\n1. **Telegram long-polling**: Must call `dropPendingUpdates(ctx, b)` before `b.Start(ctx)` and on `/restart`.\n2. **Telegram command menus**: Must register command menus simultaneously to `default`, `all_private_chats`, and `all_group_chats` via `commands.go` to prevent menu shadowing.\n3. **No UI detour**: Turns and task actions route directly to Gateway's `ChatContract`; no HTTP request may ever be sent to `archie-ui` or `internal/webui`.\n4. **Sender Confinement**: Telegram sender checks against `allowed_user_ids` must continue to fail closed before initiating any turn dispatch.\n\n## Configuration Ownership\n\n`cmd/archie-messaging` reads only:\n- `[services.gateway].target` and `[services.gatewLine truncated
},
{
"url": "/docs/prds/service-decomposition/",
"title": "Service decomposition -- decision",
"section": "PRDs",
"body": "# Service decomposition -- decision\n\n**Status:** Draft\n**Date:** 2026-09-05\n**Beads epic:** archie-core-8cda\n**Prior art:** `docs/inspiration/2026-09-05-awx-service-decomposition-and-ansible-task-model.md`,\n`docs/inspiration/module-reference-awx-decomposition-vs-archie.md`,\n`docs/inspiration/service-discovery-research-2026-09-05.md`,\n`/work/artefacts/archie-core-current-state-archify.html`,\n`/work/artefacts/archie-core-ideal-state-archify.html`\n\nThe Phase 3 UI boundary and configuration decision is ratified in\n[`ui-service-boundary.md`](/docs/prds/ui-service-boundary/). That focused document is\nthe implementation authority for `archie-core-8cda.5.2`; this parent remains\nthe broader service-distribution proposal.\n\n## Decision\n\narchie-core moves from one Go binary (`archied`) hosting every domain\nin-process to a set of independently deployable services, each exposing\nexactly one typed contract. Consumers depend on the contract; they never\nreach into another service's internal types, session state, or config.\ngRPC is the synchronous transport; NATS remains for asynchronous fan-out\n(status, telemetry) rather than being replaced.\n\nThis is a **distribution decision**, not a rewrite of Archie's domain\nmodel. The six-domain split already approved in\n`docs/architecture/index.md` and `docs/architecture/organisation.md`\n(Identity, Agent, Messaging, Workflow, Work Intake, Plugin) is unchanged;\nthis PRD is about how the *processes* running that domain logic get\npackaged, contracted, and discovered -- not about redrawing domain\nboundaries.\n\n### Target services\n\n| Service | Contract it owns | Existing code it replaces/wraps |\n|---|---|---|\n| UI Service | thin client only; talks to Gateway and State Store contracts | `internal/webui` (SPA hosting stays; API surface becomes typed contract clients, not a peer with direct access) |\n| Gateway Service | routing / turn / session contract | `internal/gateway` |\n| Messaging Service | channel-plugin contract (Telegram, email, Discord, Teams, ...) | `internal/channels` |\n| Curator Service | curator contract; **optional**, installs as a Gateway plugin | `internal/domain/curator` |\n| Scheduler Service | cron/ticker contract | `internal/domain/scheduling` |\n| Execution Environment | execution-dispatch contract | `internal/domain/workflow`, task dispatch paths |\n| Runner | isolated worktree/container execution | `internal/container`, `internal/agentexec`, `internal/worktree` |\n| State Store | state-access contract | `internal/store` |\n\n### Discovery\n\nPer `docs/inspiration/service-discovery-research-2026-09-05.md`:\n\n- **Primary:** Kubernetes-native discovery (Service + headless-Service/DNS\n + EndpointSlice, served by CoreDNS), delivered as a Helm chart + Operator,\n on k3s for single-host installs and a real cluster for multi-host or a\n hosted offering.\n- **Fallback:** NATS-based discovery (micro `$SRV` announce + JetStream KV\n registry) for no-K8s single-host installs, with a custom gRPC resolver\n reading `host:port` from the KV bucket.\n- Both implementations sit behind one internal interface:\n\n ```go\n type ServiceRegistry interface {\n Resolve(service string) []Endpoint\n Watch(service string) \u003c-chan Event // join/leave\n }\n ```\n\n- An optional service (Curator; others later) that is not installed\n must resolve to a distinct `NotInstalled` outcome, never a `Down`/\n `Unhealthy` one. Under K8s this falls out of DNS returning no record;\n under the NATS fallback it requires an explicit \"installed\" marker in the\n KV registry, not just absence-implies-down.\n\n### Distribution model this supports\n\nBoth self-hosted (customer runs the Helm chart or the NATS-fallback\nbinaries themselves) and a managed/hosted offering (we run the same Helm\nchart) are in scope -- this is the RHEL / Ansible Automation Platform\nmodel: one architecture, sold both ways. This PRD does not decide the\nbusiness model; it decides that the architecture must not foreclose\neither shape.\n\n## Why this shape\n\n`docs/inspiration/module-reference-awx-decomposition-vs-archie.md`\nestablished that Archie's existing domain split already does what AWX's\nservice-decomposition post argues for -- boundaries exist. What didn't\nexist was deployability: `internal/webui` was not a passive frontend. The\nre-baselined coupling inventory is below, and the ratified boundary is\n[`ui-service-boundary.md`](/docs/prds/ui-service-boundary/).\n\n- Telegram task actions now call Gateway's `ChatContract` directly through\n `chatTaskActorAdapter`; the old Telegram-to-webui HTTP detour is historical.\n- `internal/webui/api_chat.go` now holds `gateway.ChatContract`; direct\n `SessionStore`, `Router`, and `TurnHistory` access is historical.\n- The third coupling is closed. UI extraction shipped (`archie-core-8cda.5`,\n closed 2026-09-11): `cmd/archie-ui` serves the dashboard against remote\n Gateway and State Store contracts, holdLine truncated
},
{
"url": "/docs/prds/ui-service-boundary/",
"title": "UI Service boundary and configuration ownership",
"section": "PRDs",
"body": "# UI Service boundary and configuration ownership\n\n**Status:** Approved\n**Revision:** 1\n**Date:** 2026-09-07\n**Beads issue:** `archie-core-8cda.5.1`\n**Parent:** [`service-decomposition.md`](/docs/prds/service-decomposition/)\n**Prerequisites:** Gateway contract extraction and State Store contract ratification\n\nThis document is the authority for Phase 3 UI Service implementation. It fixes\nthe process boundary, configuration ownership, client contracts, security\nposture, readiness behavior, and migration evidence required by\n`archie-core-8cda.5.2` and later UI extraction work.\n\n## Decision\n\nThe UI Service owns the operator-facing HTTP surface and the compiled SPA. It\nserves `ui/dist` and exposes UI-specific JSON DTOs for dashboard views and\nactions. It does not own conversations, task state, workflow execution,\nchannel lifecycle, forge access, or the daemon's runtime configuration.\n\nThe UI Service consumes two typed remote capabilities:\n\n- Gateway's `gateway.ChatContract` through the versioned Gateway gRPC\n contract, for chat snapshots, messages, turns, streaming, persona changes,\n cancellation, and Gateway task actions.\n- State Store's narrow read/write contracts through the versioned State Store\n gRPC contract, for task views, events, captures, mappings, bindings, and\n configuration data explicitly projected for the dashboard.\n\nThe UI process may compose local adapters during the migration seam, but the\nadapter implements the same wire-safe contract as the remote client. The\nproduction extraction cutover removes the in-process adapter in the same\nchange; a dual-live UI authority is not permitted.\n\n## Boundary and ownership\n\n| Concern | UI Service owns | UI Service may consume | UI Service must not own or receive |\n|---|---|---|---|\n| HTTP and SPA | routes, auth middleware, UI DTOs, assets, HTTP lifecycle | contract results and health snapshots | domain entities or infrastructure handles |\n| Chat | request validation and response projection | Gateway `ChatContract` | `gateway.Router`, `SessionStore`, turn runner, model/provider runtime |\n| Task dashboard | query/action DTOs and HTTP status mapping | State Store narrow contracts and explicitly ratified task-action contract | `*store.Store`, SQL connections, workflow registry, daemon pointer |\n| Configuration page | secret-free view and update request validation | a UI configuration contract owned by the configuration/application boundary | `config.Config`, `config.Holder`, secret registry, reload implementation |\n| Readiness | UI process liveness and dependency readiness aggregation | bounded health results from Gateway and State Store | direct probing of private databases or providers |\n\nEvery value crossing the process boundary is a versioned DTO or scalar contract\nvalue. `internal/webui` may retain temporary Go-side adapters while the\nmigration is in progress, but handlers must map Gateway/State Store values to\nUI-owned response types at the HTTP edge. A constructor accepting a concrete\nGateway, State Store, daemon, or `config.Holder` is not an approved final\nboundary.\n\nThe existing UI routes remain behaviorally compatible during extraction,\nincluding task actions, SSE event delivery, config views, chat streaming,\ncapture intake, mappings, bindings, update reporting, and health endpoints.\nCompatibility preserves status codes, authentication requirements, redaction,\npagination/limits, event ordering, and failure degradation unless a separately\nratified contract changes them.\n\nThe route families have these target owners. This is the required inventory for\nthe implementation children; a route without an owner is not eligible for\ncutover.\n\n| Route family | Target owner consumed by UI | Migration status |\n|---|---|---|\n| `/api/chat/*` | Gateway `ChatContract` and its versioned client | contract seam exists; DTO and stream parity remain |\n| `/api/tasks/*`, `/api/workflows`, `/api/setup` | State Store read/action contracts plus an explicit execution/admin action contract | State Store adapter exists; UI-facing narrow surface remains |\n| `/events` | State Store `EventsSince`, polled by one UI-process pump feeding `Server.Broadcast` | RESOLVED (`archie-core-za9f`): see migration-decisions, \"Dashboard live event delivery\". No new RPC; replay cursor and drop-recovery are the existing `since` watermark in `sse.go` |\n| `/api/logs`, `/api/logs/stream` | daemon diagnostic feed, host-local | unresolved: `logging.Feed` is in-process on the daemon host and has no contract. The per-task log routes moved off this row: see `/api/tasks/*` above |\n| `/api/tasks/{id}/logs`, `/api/tasks/{id}/logs/download` | State Store task-log read contract (`TaskLogStore`: `ReadTaskLog` + `StreamTaskLogContent`), read by the UI process from the service that owns the state directory | RESOLVED (`archie-core-iaqx`): the UI process composes `TaskLogStore` from its own State Store client and serves the attemptLine truncated
}
],
"nav": [
{
"title": "Archie",
"url": "/docs/"
},
{
"title": "Architecture",
"url": "/docs/architecture/",
"items": [
{
"title": "Adversarial self-review",
"url": "/docs/architecture/adversarial-review/"
},
{
"title": "Agent System",
"url": "/docs/architecture/agent-system/"
},
{
"title": "Playbook bindings: model and threat model",
"url": "/docs/architecture/bindings/"
},
{
"title": "Configuration",
"url": "/docs/architecture/configuration/"
},
{
"title": "Dependencies and Contracts",
"url": "/docs/architecture/dependencies-and-contracts/"
},
{
"title": "Generated Documentation",
"url": "/docs/architecture/generated-documentation/"
},
{
"title": "Identity",
"url": "/docs/architecture/identity/"
},
{
"title": "Messaging and Work Intake",
"url": "/docs/architecture/messaging-and-work-intake/"
},
{
"title": "Architecture Migration Decisions",
"url": "/docs/architecture/migration-decisions/"
},
{
"title": "Task Observability",
"url": "/docs/architecture/observability/"
},
{
"title": "Repository Organisation",
"url": "/docs/architecture/organisation/"
},
{
"title": "Domain and Feature Review Procedure",
"url": "/docs/architecture/package-review/"
},
{
"title": "Plugins and Extensions",
"url": "/docs/architecture/plugins-and-extensions/"
},
{
"title": "Policy System",
"url": "/docs/architecture/policy/"
},
{
"title": "Safe Change and Recovery",
"url": "/docs/architecture/safe-change-and-recovery/"
}
]
},
{
"title": "Guides",
"items": [
{
"title": "Building your first playbook binding",
"url": "/docs/guides/first-playbook/"
}
]
},
{
"title": "PRDs",
"items": [
{
"title": "Backend-owned configuration schema for the webui -- decision",
"url": "/docs/prds/config-schema/"
},
{
"title": "Memory engine unification: four scopes, CRUD, and one write path -- decision",
"url": "/docs/prds/memory-engine-unification/"
},
{
"title": "Messaging Service boundary and transport design",
"url": "/docs/prds/messaging-service-boundary/"
},
{
"title": "Service decomposition -- decision",
"url": "/docs/prds/service-decomposition/"
},
{
"title": "UI Service boundary and configuration ownership",
"url": "/docs/prds/ui-service-boundary/"
}
]
}
]
}
+4 -17
View File
@@ -5,23 +5,10 @@ repo_url: https://github.com/samcharles93/archie-core
edit_uri: edit/main/docs/
copyright: Vectis Labs
# Everything under docs/ is published to the public internet. These subtrees are
# held back deliberately, each for its own reason:
# archive/ superseded design. Publishing it competes with the current
# architecture for a reader's attention, which is the exact
# failure this repo already fights.
# inspiration/ research notes that discuss other products by name.
# prds/ open design decisions, not settled architecture.
# github-token setup notes about credentials.
# Widen this by deleting a line; nothing else needs to change.
exclude_docs: |
archive/
inspiration/
prds/*
!prds/ui-service-boundary.md
!prds/service-decomposition.md
!prds/config-schema.md
github-token.md
# The published set and the reasons for what is held back now live in
# tools/docsite/main.go, which emits the canonical artifact a renderer reads
# (docs/data/generated/docs.json). MkDocs is being retired, so its exclusions are
# not duplicated here: a second copy of a decision is a second thing to keep true.
# The generated release-notes pages are linked from docs/news/index.md and stay
# published, but they are kept out of the derived navigation: a version list
+14 -6
View File
@@ -33,13 +33,21 @@ const (
modeCheck
)
// generatedArtifacts declares the directories docsgen owns end-to-end, and the
// file names it produces in each. Obsolete detection is scoped to these
// directories: an arbitrary --out location is not ours to judge, and the target
// layout anticipates further named artifacts here (catalog.json, AsyncAPI
// output), which must not be reported as stale merely for existing.
// generatedArtifacts declares the directories whose contents are generated
// end-to-end, and every file name accounted for in each. Obsolete detection is
// scoped to these directories: an arbitrary --out location is not ours to judge,
// and the target layout anticipates further named artifacts here (catalog.json,
// AsyncAPI output), which must not be reported as stale merely for existing.
//
// docs/data/generated is shared. docsgen writes contracts.json from the
// Go contract types; tools/docsite writes docs.json from the Markdown sources.
// Each tool verifies its OWN file, and both files are named here because this map
// is the directory's ownership record: whichever tool runs first must not report
// the other's artifact as an unaccounted file. docsgen does not verify docs.json,
// and tools/docsite keeps no registry of its own for that reason -- one fact, one
// home.
var generatedArtifacts = map[string][]string{
"docs/data/generated": {"contracts.json"},
"docs/data/generated": {"contracts.json", "docs.json"},
}
// options is a parsed command line.
+599
View File
@@ -0,0 +1,599 @@
// Command docsite derives the public documentation set into one canonical,
// host-neutral artifact.
//
// The Markdown is the source; docs/data/generated/docs.json is the fact set every
// renderer reads. MkDocs was the first renderer and the Astro app is the second,
// so the artifact is the product and the Markdown stays the source --
// tools/newsgen states the same principle for releases, where the JSON is
// canonical and the Markdown is an adapter.
//
// The published set is declared here rather than in a site configuration, because
// the exclusions are decisions rather than details: see publishRules. A renderer
// that reads this artifact needs no copy of them.
//
// Output is generated and committed:
//
// docs/data/generated/docs.json
//
// `docsite check` regenerates the artifact in a temporary directory and fails when
// the committed one differs, without touching the tree.
//
// This tool keeps no registry of the generated directory it writes into: that
// directory is accounted for by docsgen's generatedArtifacts, which names this
// file, so a second registry here would be a second home for one fact.
package main
import (
"bytes"
"encoding/json"
"errors"
"flag"
"fmt"
"io/fs"
"log"
"os"
"path"
"path/filepath"
"regexp"
"sort"
"strings"
"unicode"
)
// artifactPath is the committed artifact this tool owns, relative to the
// repository root.
const artifactPath = "docs/data/generated/docs.json"
// docsDir is the Markdown source tree, relative to the repository root.
const docsDir = "docs"
// urlPrefix is the path every page is served under. The directory-style paths
// MkDocs produced are kept exactly, so existing links keep working.
const urlPrefix = "/docs/"
type mode int
const (
modeWrite mode = iota
modeCheck
)
// options is a parsed command line.
type options struct {
mode mode
repoRoot string
}
func main() {
parsed, err := parseOptions(os.Args[1:])
if err != nil {
log.Fatalf("docsite: %v", err)
}
if parsed.mode == modeCheck {
err = check(parsed.repoRoot)
} else {
err = write(parsed.repoRoot)
}
if err != nil {
log.Fatalf("docsite: %v", err)
}
}
// parseOptions resolves the optional `check` subcommand and the flags that may
// follow it. The subcommand is peeled off before flag parsing rather than left for
// it to ignore: `docsite --repo-root .. check` would otherwise fall through to the
// write path and rewrite the artifact check promises not to touch.
func parseOptions(args []string) (options, error) {
parsed := options{repoRoot: "."}
if len(args) > 0 && args[0] == "check" {
parsed.mode = modeCheck
args = args[1:]
}
flags := flag.NewFlagSet("docsite", flag.ContinueOnError)
flags.StringVar(&parsed.repoRoot, "repo-root", ".", "repository root to read docs from")
if err := flags.Parse(args); err != nil {
return options{}, err
}
if flags.NArg() > 0 {
return options{}, fmt.Errorf("unexpected argument %q", flags.Arg(0))
}
return parsed, nil
}
// document is the artifact. Its keys are a frozen contract: a renderer is built
// against them, so a rename here breaks a consumer this repository does not
// compile.
type document struct {
Pages []page `json:"pages"`
Nav []navNode `json:"nav"`
}
// page is one published document. Body is the raw Markdown source: presentation
// belongs to the renderer, so this artifact deliberately does not carry HTML.
type page struct {
URL string `json:"url"`
Title string `json:"title"`
Section string `json:"section"`
Body string `json:"body"`
}
// navNode is one entry in the sidebar: a link when URL is set, a group when Items
// is.
type navNode struct {
Title string `json:"title"`
URL string `json:"url,omitempty"`
Items []navNode `json:"items,omitempty"`
}
// publishRules is the published set, and the reasons, carried from the site
// configuration this replaces. Everything under docs/ is public except:
//
// - archive/ superseded design. Publishing it competes with the current
// architecture for a reader's attention, which is the exact failure this
// repository already fights.
// - inspiration/ research notes that discuss other products by name.
// - prds/ open design decisions rather than settled architecture. Two
// kinds of PRD are published anyway, and for different reasons:
// · ui-service-boundary.md, service-decomposition.md and config-schema.md
// are ratified and read as reference.
// · memory-engine-unification.md and messaging-service-boundary.md are
// cited by documents of record -- settled architecture and the migration
// register -- so excluding them withheld nothing a reader was spared: it
// broke citations. A blanket prds/* rule had left those links 404 on the
// published site, which is worse than either publishing or not linking.
// - news/ release notes, which tools/newsgen owns and a renderer reads
// from releases.json.
// - github-token.md setup notes about credentials.
//
// Widening the set means editing this list. The comment is the record: a renderer
// that reads the artifact cannot see why a page is absent, and neither can the
// next person deciding whether a citation should publish its target.
var (
heldBackTrees = map[string]bool{"archive": true, "inspiration": true, "news": true}
publishedPRDs = map[string]bool{
"ui-service-boundary.md": true,
"service-decomposition.md": true,
"config-schema.md": true,
"memory-engine-unification.md": true,
"messaging-service-boundary.md": true,
}
heldBackPages = map[string]bool{"github-token.md": true}
)
// heldBack reports whether one path relative to docs/ is deliberately absent from
// the published set. Links and pages are judged by this one rule, so a page is
// never published while a link to it is refused, or the reverse.
func heldBack(relative string) bool {
if heldBackPages[relative] {
return true
}
segments := strings.Split(relative, "/")
if heldBackTrees[segments[0]] {
return true
}
if segments[0] == "prds" && len(segments) > 1 && !publishedPRDs[strings.Join(segments[1:], "/")] {
return true
}
return false
}
// published reports whether one path relative to docs/ is a page the artifact
// carries.
func published(relative string) bool {
return strings.HasSuffix(relative, ".md") && !heldBack(relative)
}
// intraDocsLink matches a Markdown inline link or image. The target is the only
// part this tool rewrites; everything else in the match is preserved.
var intraDocsLink = regexp.MustCompile(`\]\(\s*([^)\s]+)`)
// unresolvedLink names a link the artifact cannot honestly carry: its target is a
// page or asset the published set holds back. It is reported rather than rewritten
// to a hopeful URL or dropped, because such a link exists only because a site build
// used to publish the whole tree -- it is exactly what the exclusions break, and
// the reader deserves the list rather than a guess.
type unresolvedLink struct {
File string
Line int
Target string
// Why names what is wrong, because the two cases need different repairs: a
// target the set holds back is published or unlinked, and a fragment that
// matches no heading is a broken citation to fix in the prose.
Why string
}
func (u unresolvedLink) Error() string {
return fmt.Sprintf("%s:%d: %s: %s", u.File, u.Line, u.Target, u.Why)
}
// Reasons a link cannot be carried.
const (
whyTargetAbsent = "its target is not in the published set"
whyAnchorAbsent = "the target page has no such heading"
)
// headingSlugs lists the ids a page's headings produce, by the same rule a
// Markdown renderer's table of contents uses: lowercase, punctuation dropped,
// whitespace runs collapsed to one hyphen, and a repeated heading suffixed in
// order. An explicit {#id} wins, because that is what it is for.
func headingSlugs(body string) map[string]bool {
slugs := map[string]bool{}
counts := map[string]int{}
for line := range strings.Lines(body) {
trimmed := strings.TrimSpace(line)
hashes := 0
for hashes < len(trimmed) && trimmed[hashes] == '#' {
hashes++
}
if hashes == 0 || hashes == len(trimmed) || trimmed[hashes] != ' ' {
continue
}
title := strings.TrimSpace(trimmed[hashes:])
if open := strings.LastIndex(title, "{#"); open >= 0 && strings.HasSuffix(title, "}") {
slugs[strings.TrimSuffix(title[open+2:], "}")] = true
continue
}
slug := slugify(title)
if slug == "" {
continue
}
if counts[slug] > 0 {
slug = fmt.Sprintf("%s_%d", slug, counts[slug])
}
counts[strings.SplitN(slug, "_", 2)[0]]++
slugs[slug] = true
}
return slugs
}
// slugify is the table-of-contents id for one heading.
func slugify(title string) string {
var b strings.Builder
pending := false
for _, r := range strings.ToLower(title) {
switch {
case unicode.IsLetter(r) || unicode.IsDigit(r) || r == '-' || r == '_':
if pending && b.Len() > 0 {
b.WriteByte('-')
}
pending = false
b.WriteRune(r)
case unicode.IsSpace(r):
pending = true
}
}
return b.String()
}
// linkResolver rewrites one page's intra-docs links to the urls the artifact
// carries. It uses the same path-to-url rule as the pages themselves, so a link
// and the page it points at cannot disagree; only the anchor is carried over
// unchanged.
func linkResolver(relative string, urls map[string]string, slugs map[string]map[string]bool, sourceDir string) func(string) (string, string) {
dir := path.Dir(relative)
return func(target string) (string, string) {
if target == "" {
return "", whyTargetAbsent
}
// Anchors and anything with a scheme are not this tool's to resolve.
if strings.HasPrefix(target, "#") || strings.Contains(target, "://") ||
strings.HasPrefix(target, "mailto:") || strings.HasPrefix(target, "//") {
return target, ""
}
linkPath, anchor, _ := strings.Cut(target, "#")
fragment := ""
if anchor != "" {
fragment = "#" + anchor
}
resolved := path.Clean(path.Join(dir, linkPath))
if strings.HasSuffix(resolved, ".md") {
url, ok := urls[resolved]
if !ok {
return "", whyTargetAbsent
}
// A fragment that matches no heading lands the reader at the top of a
// long document with no indication why, so it is a finding rather than
// a pass: the target resolving is not the same as the citation working.
if anchor != "" && !slugs[resolved][anchor] {
return "", whyAnchorAbsent
}
return url + fragment, ""
}
// An asset is served at its own path, when the set does not hold it back
// and it is actually there.
if !heldBack(resolved) {
if _, err := os.Stat(filepath.Join(sourceDir, filepath.FromSlash(resolved))); err == nil {
return urlPrefix + resolved + fragment, ""
}
}
return "", whyTargetAbsent
}
}
// resolveLinks rewrites every intra-docs link in body and reports the ones whose
// target the published set holds back.
func resolveLinks(relative, body string, urls map[string]string, slugs map[string]map[string]bool, sourceDir string) (string, []unresolvedLink) {
resolve := linkResolver(relative, urls, slugs, sourceDir)
var unresolved []unresolvedLink
lines := strings.Split(body, "\n")
for index, line := range lines {
lines[index] = intraDocsLink.ReplaceAllStringFunc(line, func(match string) string {
target := strings.TrimSpace(strings.TrimPrefix(match, "]("))
replacement, why := resolve(target)
if why != "" {
unresolved = append(unresolved, unresolvedLink{File: relative, Line: index + 1, Target: target, Why: why})
return match
}
return "](" + replacement
})
}
return strings.Join(lines, "\n"), unresolved
}
// load reads every published page and derives the sidebar from the layout, which
// is how MkDocs derived it: a page added under a directory appears without a
// second edit anywhere.
func load(repoRoot string) (document, error) {
sourceDir := filepath.Join(repoRoot, docsDir)
bodies := map[string]string{}
err := filepath.WalkDir(sourceDir, func(filename string, entry fs.DirEntry, err error) error {
if err != nil {
return err
}
if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".md") {
return nil
}
relative, err := filepath.Rel(sourceDir, filename)
if err != nil {
return err
}
relative = filepath.ToSlash(relative)
if !published(relative) {
return nil
}
body, err := os.ReadFile(filename)
if err != nil {
return fmt.Errorf("read %s: %w", relative, err)
}
bodies[relative] = string(body)
return nil
})
if err != nil {
return document{}, fmt.Errorf("walk %s: %w", docsDir, err)
}
if len(bodies) == 0 {
return document{}, fmt.Errorf("no published pages under %s", docsDir)
}
// Every page's url first, so a link resolves against the whole set rather than
// against whatever the walk had reached.
urls := make(map[string]string, len(bodies))
slugs := make(map[string]map[string]bool, len(bodies))
for relative, body := range bodies {
urls[relative] = pageURL(relative)
slugs[relative] = headingSlugs(body)
}
var doc document
var unresolved []unresolvedLink
for relative, body := range bodies {
resolved, missing := resolveLinks(relative, body, urls, slugs, sourceDir)
unresolved = append(unresolved, missing...)
doc.Pages = append(doc.Pages, page{
URL: urls[relative],
Title: pageTitle(relative, body),
Section: sectionFor(relative),
Body: resolved,
})
}
if len(unresolved) > 0 {
sort.Slice(unresolved, func(i, j int) bool {
if unresolved[i].File != unresolved[j].File {
return unresolved[i].File < unresolved[j].File
}
return unresolved[i].Line < unresolved[j].Line
})
problems := make([]string, 0, len(unresolved))
for _, link := range unresolved {
problems = append(problems, link.Error())
}
return document{}, fmt.Errorf(
"%d link(s) cannot be carried; see each reason:\n %s",
len(unresolved), strings.Join(problems, "\n "))
}
sort.Slice(doc.Pages, func(i, j int) bool { return doc.Pages[i].URL < doc.Pages[j].URL })
doc.Nav = navFor(doc.Pages)
return doc, nil
}
// pageURL is the directory-style path a static site produces for one source file:
// docs/index.md is the root, docs/a/index.md is the section's own landing page,
// and docs/a/b.md is a page inside it.
func pageURL(relative string) string {
if relative == "index.md" {
return urlPrefix
}
dir, file := path.Split(relative)
if file == "index.md" {
return urlPrefix + dir // path.Split keeps the trailing slash
}
return urlPrefix + dir + strings.TrimSuffix(file, ".md") + "/"
}
// pageTitle is the page's own first heading, which is the title its author chose,
// falling back to its file name so a page without one is never nameless.
func pageTitle(relative, body string) string {
for line := range strings.Lines(body) {
if heading, ok := strings.CutPrefix(strings.TrimSpace(line), "# "); ok {
if title := strings.TrimSpace(heading); title != "" {
return title
}
}
}
return readableName(strings.TrimSuffix(path.Base(relative), ".md"))
}
// sectionFor is the group a page belongs to: its first directory, titled. A page
// directly under docs/ has no section, because it is not in one.
func sectionFor(relative string) string {
segments := strings.Split(relative, "/")
if len(segments) < 2 {
return ""
}
return readableName(segments[0])
}
// acronyms are directory names whose natural label is not title case, so a
// sidebar does not read "Prds".
var acronyms = map[string]string{"prds": "PRDs"}
// readableName turns a directory or file name into a label.
func readableName(name string) string {
if label, ok := acronyms[name]; ok {
return label
}
words := strings.FieldsFunc(name, func(r rune) bool { return r == '-' || r == '_' })
for i, word := range words {
words[i] = strings.ToUpper(word[:1]) + word[1:]
}
return strings.Join(words, " ")
}
// navFor derives the sidebar tree from the pages. At each level the landing page
// comes first and the rest follow alphabetically, so a reader meets the section's
// own page before its children -- the order a static site's navigation uses.
func navFor(pages []page) []navNode {
type node struct {
title string
url string
children map[string]*node
}
root := &node{children: map[string]*node{}}
for _, entry := range pages {
segments := strings.Split(strings.Trim(strings.TrimPrefix(entry.URL, urlPrefix), "/"), "/")
current := root
for _, segment := range segments {
if segment == "" {
continue
}
child, ok := current.children[segment]
if !ok {
child = &node{title: readableName(segment), children: map[string]*node{}}
current.children[segment] = child
}
current = child
}
if current == root {
// The root index: the site's home, listed at the top of the sidebar.
root.title, root.url = entry.Title, entry.URL
continue
}
current.url, current.title = entry.URL, entry.Title
}
var render func(*node) []navNode
render = func(parent *node) []navNode {
names := make([]string, 0, len(parent.children))
for name := range parent.children {
names = append(names, name)
}
sort.Strings(names)
// A directory whose own index is published: the section's landing page
// leads with its link, then its children.
var nodes []navNode
for _, name := range names {
child := parent.children[name]
entry := navNode{Title: child.title}
if child.url != "" {
entry.URL = child.url
}
entry.Items = render(child)
// A directory is a group whether it holds one page or twenty: the tree
// mirrors the layout a reader sees, and collapsing a one-page directory
// into a bare link would be a shape this artifact invented rather than
// one it reports.
nodes = append(nodes, entry)
}
return nodes
}
var nav []navNode
if root.url != "" {
nav = append(nav, navNode{Title: root.title, URL: root.url})
}
return append(nav, render(root)...)
}
// marshal renders the artifact. Indented and newline-terminated so a diff reads
// as a change to a page rather than a change to one line.
func marshal(doc document) ([]byte, error) {
encoded, err := json.MarshalIndent(doc, "", " ")
if err != nil {
return nil, fmt.Errorf("encode artifact: %w", err)
}
return append(encoded, '\n'), nil
}
// write regenerates the artifact, leaving the file untouched when its bytes
// already match so a no-op run does not churn the tree.
func write(repoRoot string) error {
doc, err := load(repoRoot)
if err != nil {
return err
}
encoded, err := marshal(doc)
if err != nil {
return err
}
target := filepath.Join(repoRoot, artifactPath)
existing, err := os.ReadFile(target)
if err == nil && bytes.Equal(existing, encoded) {
fmt.Printf("docsite: %s is up to date\n", artifactPath)
return nil
}
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return fmt.Errorf("read %s: %w", artifactPath, err)
}
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
return fmt.Errorf("create %s: %w", filepath.Dir(artifactPath), err)
}
if err := os.WriteFile(target, encoded, 0o644); err != nil {
return fmt.Errorf("write %s: %w", artifactPath, err)
}
fmt.Printf("docsite: wrote %s (%d pages)\n", artifactPath, len(doc.Pages))
return nil
}
// check regenerates into a temporary directory and compares, so a source edit that
// has not been reflected in the committed artifact fails without the tree being
// touched.
func check(repoRoot string) error {
tmpDir, err := os.MkdirTemp("", "docsite-check-")
if err != nil {
return fmt.Errorf("create temporary directory: %w", err)
}
defer func() { _ = os.RemoveAll(tmpDir) }()
generated, err := load(repoRoot)
if err != nil {
return err
}
expected, err := marshal(generated)
if err != nil {
return err
}
committed, err := os.ReadFile(filepath.Join(repoRoot, artifactPath))
if errors.Is(err, fs.ErrNotExist) {
return fmt.Errorf("%s is missing; regenerate it by running `go -C tools run ./docsite --repo-root ..` from the tools module", artifactPath)
}
if err != nil {
return fmt.Errorf("read %s: %w", artifactPath, err)
}
if bytes.Equal(expected, committed) {
fmt.Printf("docsite: %s is up to date (%d pages)\n", artifactPath, len(generated.Pages))
return nil
}
return fmt.Errorf("%s is stale; regenerate it by running `go -C tools run ./docsite --repo-root ..` from the tools module", artifactPath)
}
+261
View File
@@ -0,0 +1,261 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
)
// TestPublishedSetIsTheDecision pins the count against the real tree. The number
// is the decision this tool carries: everything under docs/ is public except five
// named exclusions, so a page added or removed on purpose has to be
// acknowledged here rather than sliding into the published site.
func TestPublishedSetIsTheDecision(t *testing.T) {
doc, err := load("../..")
if err != nil {
t.Fatalf("load: %v", err)
}
if len(doc.Pages) != 23 {
titles := make([]string, 0, len(doc.Pages))
for _, entry := range doc.Pages {
titles = append(titles, entry.URL)
}
t.Fatalf("pages = %d, want the 21 the published set declares:\n %s", len(doc.Pages), strings.Join(titles, "\n "))
}
published := map[string]bool{}
for _, entry := range doc.Pages {
published[entry.URL] = true
if entry.Body == "" {
t.Errorf("%s carries no body; the artifact is the source a renderer needs", entry.URL)
}
if !strings.HasPrefix(entry.Body, "#") {
t.Errorf("%s body does not start with a heading: %q", entry.URL, firstLine(entry.Body))
}
}
for _, held := range []string{"/docs/archive/", "/docs/inspiration/", "/docs/news/", "/docs/github-token/"} {
for url := range published {
if strings.HasPrefix(url, held) {
t.Errorf("%s is published; it is held back deliberately", url)
}
}
}
for _, want := range []string{
"/docs/",
"/docs/architecture/organisation/",
"/docs/prds/ui-service-boundary/",
"/docs/prds/service-decomposition/",
"/docs/prds/config-schema/",
// Cited by documents of record, so a blanket prds/* exclusion had left
// these links 404 on the published site.
"/docs/prds/memory-engine-unification/",
"/docs/prds/messaging-service-boundary/",
} {
if !published[want] {
t.Errorf("%s is not published; the set declares it", want)
}
}
// An unpublished PRD is the common case, so one absence is asserted to keep the
// rule from being satisfied by publishing everything.
if published["/docs/prds/curator-sampler-wave1/"] {
t.Error("an open PRD is published; prds/ is held back except the ratified and cited ones")
}
}
func firstLine(body string) string {
line := body
if index := strings.IndexByte(body, '\n'); index >= 0 {
line = body[:index]
}
return line
}
// TestPageURLsAreTheStaticSitePaths: every existing link points at the
// directory-style path a static site produces, so the artifact has to reproduce
// it exactly rather than inventing a cleaner one.
func TestPageURLsAreTheStaticSitePaths(t *testing.T) {
for _, tc := range []struct{ relative, want string }{
{"index.md", "/docs/"},
{"architecture/index.md", "/docs/architecture/"},
{"architecture/organisation.md", "/docs/architecture/organisation/"},
{"prds/config-schema.md", "/docs/prds/config-schema/"},
{"guides/nested/deep.md", "/docs/guides/nested/deep/"},
} {
if got := pageURL(tc.relative); got != tc.want {
t.Errorf("pageURL(%q) = %q, want %q", tc.relative, got, tc.want)
}
}
}
// TestNavMirrorsTheLayout: the tree is derived from the directory structure, so a
// page added under a directory appears in the sidebar without a second edit -- the
// property MkDocs had, and the one a hand-maintained nav loses.
func TestNavMirrorsTheLayout(t *testing.T) {
nav := navFor([]page{
{URL: "/docs/", Title: "Archie"},
{URL: "/docs/architecture/", Title: "Architecture"},
{URL: "/docs/architecture/organisation/", Title: "Organisation"},
{URL: "/docs/architecture/policy/", Title: "Policy"},
{URL: "/docs/guides/first-playbook/", Title: "Building your first playbook binding"},
})
if len(nav) != 3 {
t.Fatalf("nav = %+v, want home plus two groups", nav)
}
if nav[0].URL != "/docs/" {
t.Errorf("first entry = %+v, want the home page", nav[0])
}
architecture := nav[1]
if architecture.Title != "Architecture" || architecture.URL != "/docs/architecture/" {
t.Errorf("group = %+v, want the section with its own landing page", architecture)
}
if len(architecture.Items) != 2 || architecture.Items[0].Title != "Organisation" {
t.Errorf("items = %+v, want the section's pages in order", architecture.Items)
}
// A directory holding one page is still a group: collapsing it would be a
// shape this artifact invented rather than one it reports.
guides := nav[2]
if guides.Title != "Guides" || len(guides.Items) != 1 || guides.Items[0].URL != "/docs/guides/first-playbook/" {
t.Errorf("guides = %+v, want a group wrapping its single page", guides)
}
}
// TestCheckFailsOnAStaleArtifactAndLeavesTheTreeAlone: check is what runs in the
// gate, so it has to fail on a source edit that never reached the artifact without
// writing anything itself.
func TestCheckFailsOnAStaleArtifactAndLeavesTheTreeAlone(t *testing.T) {
root := t.TempDir()
writeFile(t, filepath.Join(root, "docs", "index.md"), "# Archie\n\nHello.\n")
writeFile(t, filepath.Join(root, "docs", "architecture", "policy.md"), "# Policy\n\nRules.\n")
writeFile(t, filepath.Join(root, "docs", "archive", "old.md"), "# Old\n\nSuperseded.\n")
stale := filepath.Join(root, "docs", "data", "generated", "docs.json")
writeFile(t, stale, "{}\n")
if err := check(root); err == nil {
t.Fatal("check(stale) = nil, want a failure: the committed artifact disagrees with the sources")
}
before, err := os.ReadFile(stale)
if err != nil {
t.Fatal(err)
}
if string(before) != "{}\n" {
t.Fatalf("check rewrote the artifact: %q", before)
}
if err := write(root); err != nil {
t.Fatalf("write: %v", err)
}
if err := check(root); err != nil {
t.Fatalf("check after write = %v, want it to pass", err)
}
doc, err := load(root)
if err != nil {
t.Fatal(err)
}
if len(doc.Pages) != 2 {
t.Fatalf("pages = %d, want the two published pages (archive/ is held back)", len(doc.Pages))
}
}
func writeFile(t *testing.T, path, content string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
// TestLinksResolveInsideTheSetAndFailOutside pins both halves of link handling.
// An intra-docs link becomes the target page's url, using the same rule that
// builds the target's url, and anchors and scheme-qualified links pass through. A
// link whose target the set holds back fails the load, naming the file, the line
// and the target: that refusal is what made a blanket prds/* exclusion visible,
// because three citations from settled architecture had been 404s on the published
// site, and a hopeful rewrite would have hidden them.
func TestLinksResolveInsideTheSetAndFailOutside(t *testing.T) {
root := t.TempDir()
writeFile(t, filepath.Join(root, "docs", "organisation.md"), "# Organisation\n\nBack [home](index.md).\n")
writeFile(t, filepath.Join(root, "docs", "archive", "old.md"), "# Old\n")
writeFile(t, filepath.Join(root, "docs", "index.md"),
"# Archie\n\nSee [Organisation](organisation.md#organisation).\n\n[Old](archive/old.md) is held back.\n")
_, err := load(root)
if err == nil {
t.Fatal("load = nil, want a refusal: a published page links into archive/, which the set holds back")
}
for _, want := range []string{"index.md:5", "archive/old.md", "not in the published set"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("error = %q, want it to contain %q", err, want)
}
}
// The same tree without the out-of-set link: the in-set link becomes the target
// page's url, and the anchor rides along.
writeFile(t, filepath.Join(root, "docs", "index.md"),
"# Archie\n\nSee [Organisation](organisation.md#organisation), [OpenAI](https://openai.com/) and [below](#anchors).\n")
doc, err := load(root)
if err != nil {
t.Fatalf("load: %v", err)
}
var index string
for _, entry := range doc.Pages {
if entry.URL == "/docs/" {
index = entry.Body
}
}
for _, want := range []string{
"(/docs/organisation/#organisation)",
"(https://openai.com/)",
"(#anchors)",
} {
if !strings.Contains(index, want) {
t.Errorf("resolved body = %q, want it to contain %q", index, want)
}
}
}
// TestFragmentsAreValidatedAgainstTheTargetsHeadings: a link can resolve to a page
// and still be broken, because a fragment that matches no heading lands the reader
// at the top of a long document with no indication why. The slug rule mirrors a
// Markdown renderer's table of contents, so the ids it accepts are the ids the
// rendered page carries.
func TestFragmentsAreValidatedAgainstTheTargetsHeadings(t *testing.T) {
root := t.TempDir()
writeFile(t, filepath.Join(root, "docs", "decisions.md"),
"# Decisions\n\n## 5. Memory placement and storage\n\n## Bindings & secrets\n")
writeFile(t, filepath.Join(root, "docs", "index.md"),
"# Archie\n\n[Good](decisions.md#5-memory-placement-and-storage).\n\n[Bad](decisions.md#5).\n")
_, err := load(root)
if err == nil {
t.Fatal("load = nil, want a refusal: #5 matches no heading on the target page")
}
for _, want := range []string{"index.md:5", "#5", "no such heading"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("error = %q, want it to contain %q", err, want)
}
}
if strings.Contains(err.Error(), "#5-memory-placement-and-storage") {
t.Errorf("error = %q, want only the broken fragment named", err)
}
// A fragment that does match, including one whose heading has punctuation and an
// ampersand, passes and is carried verbatim.
writeFile(t, filepath.Join(root, "docs", "index.md"),
"# Archie\n\n[Good](decisions.md#5-memory-placement-and-storage).\n\n[Also](decisions.md#bindings-secrets).\n")
doc, err := load(root)
if err != nil {
t.Fatalf("load: %v", err)
}
for _, entry := range doc.Pages {
if entry.URL != "/docs/" {
continue
}
if !strings.Contains(entry.Body, "(/docs/decisions/#bindings-secrets)") {
t.Errorf("resolved body = %q, want the ampersand heading's slug", entry.Body)
}
}
}