docs(prds): land the memory engine unification decision

Approved for implementation, dated 2026-09-16, previously untracked.
Settles four open items in docs/architecture/migration-decisions.md#5 and
supersedes the memory-turn-context.md draft.

Section 8's deletion list no longer names config.Memory.SessionTTL: the
council carried that deletion standalone and f25dd1e removed the field,
so listing it here would schedule work already done. Provider and
ProviderConfig are unchanged, both still present.
This commit is contained in:
sam committed 2026-09-17 00:34:01 +10:00
1 parent f25dd1e867
commit 592a6262b5
1 file changed
+362
+362
View File
@@ -0,0 +1,362 @@
# Memory engine unification: four scopes, CRUD, and one write path -- decision
**Status:** Approved for implementation
**Date:** 2026-09-16
**Findings:** `memory-4` and its dependants in the adversarial sweep
(`.local/issue-tracker`); the real defect is the split described below, for
which `memory-4` is a symptom
**Supersedes:** the draft `memory-turn-context.md` (its synchronous-read
argument survives and is reused; its `Manager` method and consumer-owned port do
not)
**Settles:** four open items in `docs/architecture/migration-decisions.md#5`
(listed at the end)
## Problem
There are **two live memory stores for one assistant**, split by which component
writes them, and the chat turn reads neither.
| Family | Written by | Root | Format |
|---|---|---|---|
| `internal/memory` (`Manager`, `builtin.Provider`) | the chat agent's `memory_edit` tool via `memorytoolprovider` (`bootstrap.go:1198`) | `<workDir>/memory` | `MEMORY.md` / `USER.md` |
| `internal/domain/memory` + `internal/infrastructure/memory` | the session curator (`sessioncurator.go:163`) | `<workDir>/memory-engine` | identity-keyed blocks |
They share no code, no format and no scope vocabulary. Neither reaches a prompt:
`Manager.SystemPromptBlock` (`internal/memory/manager.go:429`) has no production
caller anywhere, so the agent can write memory through a tool and never read it
back.
Three further defects fall out of the same split:
1. **The engine's contract is not CRUD.** `Store` is `Write`/`Query`/`List`/
`Forget` — create, read, delete, and **no Update**. The retired file provider
*does* have `handleAdd`/`handleReplace`/`handleRemove`, so the family being
deleted is operationally richer than its replacement.
2. **There is no scope model.** The contract carries one opaque `Identity
string`, and the curator fills it with a **session id** (`sessioncurator.go:164`)
— which is none of the four scopes `docs/architecture/agent-system.md:69-78`
requires, and stops meaning anything when the session ends.
3. **The chat turn has no user identity on half the channels.** `SenderID` is
populated by Telegram and email (a person) and by webhook (a **route path**,
`webhook.go:179`), and not at all by the dashboard, whose gateway wire message
carries no sender field. It is also never persisted: `sender_id` appears
nowhere in `session_store_sqlite.go`.
`agent-system.md:100` says the write-scope selection and cross-scope sharing
rules "require a focused memory design decision". This is that decision.
## Decision
### 1. One engine, addressed by four typed scopes
`internal/domain/memory` is the only memory engine. `Observation.Identity`,
`Query.Identity`, `Record.Identity` and `List(identity)` are deleted, not
extended: a field whose meaning depends on the caller cannot carry a scope.
```go
type ScopeKind string
const (
ScopeGlobal ScopeKind = "global" // shared across the application
ScopeAgent ScopeKind = "agent" // private to one Agent
ScopeUser ScopeKind = "user" // one user, across their Agents
ScopeAgentUser ScopeKind = "agent-user" // one Agent's relationship with one user
)
type Scope struct {
Kind ScopeKind
Agent AgentID // required for ScopeAgent, ScopeAgentUser
User IdentityID // required for ScopeUser, ScopeAgentUser
}
func (s Scope) Validate() error // rejects a component present where it is not required
func (s Scope) Key() string // canonical, collision-free storage key
```
`AgentID` and `IdentityID` are opaque. This package never resolves or
canonicalises them.
A turn's resolved identity is one value:
```go
type Subject struct {
AgentID AgentID
UserID IdentityID // "" when the channel carries no per-person identity
}
func (s Subject) Scopes() []Scope // readable, most specific first
func (s Subject) WritableScopes() []Scope // never includes ScopeGlobal
```
**Access control is naming a scope, and nothing else.** The read path names
exactly the scopes of the resolved `Subject`; the engine never unions beyond what
it is handed and applies no policy of its own. An engine that enforced policy
would need to know about channels, sessions and bindings, and every backend would
reimplement it. "Agent-wide is private to its owning Agent by default"
(`agent-system.md:80-81`) falls out mechanically, because `ScopeAgent{a}` and
`ScopeAgent{b}` are different storage keys.
`ScopeGlobal` is never model-writable. Writing it is an operator action; nothing
produces one yet, but the contract models it because the read set includes it.
### 2. CRUD, with revisions retained
```go
type Store interface {
Create(ctx context.Context, in NewRecord) (Record, error)
Get(ctx context.Context, scope Scope, id RecordID) (Record, error) // ErrNotFound
Query(ctx context.Context, q Query) ([]Record, error) // heads only
List(ctx context.Context, scope Scope) ([]Record, error) // heads only
Update(ctx context.Context, in RecordUpdate) (Record, error) // ErrNotFound, ErrStaleRevision
Forget(ctx context.Context, scope Scope, id RecordID) error // idempotent
Revisions(ctx context.Context, scope Scope, id RecordID) ([]Revision, error)
}
```
`Get` and `Forget` take a scope. The current `Forget` scans every identity's
store to find a marker (`builtin_engine.go:189-210`) purely because the contract
gave it nothing better.
`Record` carries what `agent-system.md:97-99` requires be retained: `Scope`,
`Kind`, `Content`, `Revision`, `Author`, `OriginUser`, `Source`, `CreatedAt`,
`UpdatedAt`. `Metadata` is dropped — nothing sets it and the parser never
restores it, so it is currently a field that silently does not round-trip.
`Update` supersedes rather than overwrites. The superseded state is appended to
the scope's `HISTORY.md` **before** the live block is replaced, so a crash
between the two leaves an extra history entry and an unchanged record —
recoverable — never a record whose previous state was lost. `Update` carries no
`Scope`: a record may never move between scopes, and the API makes that
unrepresentable rather than validating it. `Expected` optionally names the
revision being replaced and returns `ErrStaleRevision` on mismatch, which matters
because two processes (`archied` and `archie-gateway`) can hold the same scope's
file.
`Forget` records a deleted revision before removing the live block, so a record's
provenance outlives its content.
### 3. The caller resolves identity; the engine never does
Resolution happens in the gateway turn path, where the native sender id and the
session are both in hand, and it is supplied **per channel by the composition**:
```go
// TurnRunnerConfig
// UserIdentity resolves the initiating user's identity for one inbound message.
// false means the channel carries no per-person identity, and the turn gets
// global and agent scopes only. Nil means the same.
UserIdentity func(msg messaging.Message) (IdentityID, bool)
```
This must be injectable rather than a blanket `channel + ":" + SenderID`
concatenation, because a **webhook's `SenderID` is a route path**
(`webhook.go:179`), not a person; treating it as one would give a URL an
identity. Telegram and email supply their sender id; the dashboard and webhook
resolve nothing.
**Fail closed.** When `UserID` is empty the turn proceeds with global and agent
scopes only, and a write targeting `user` or `agent-user` fails with a clear
error. `agent-system.md:92-94` forbids showing one user's memory to another, so
"we do not know who this is" must mean "show none of it", never "fall back to a
wider scope".
### 4. The chat read path
Synchronous and total. It runs inside `prepareTurn` (`turn.go:287`) before the
prompt is built, never returns an error to the caller, and degrades to an empty
block on any engine failure or provider panic. A fire-and-forget hook would race
the prompt build in the same turn, which is why this is not a lifecycle hook.
The read is a scope-only `Query` over the subject's scopes — **not** ranked
search, which remains open in `migration-decisions.md#5`. A ranked read on the
hot path with no ranking implementation would be a lie.
`SystemPromptConfig` gains a `Memory` field, rendered into
`templates/archie.md.tpl` between `<tools>` (line 41) and `<env>` (line 50):
```
<memory purpose="durable_context" trust="data">
```
`trust="data"` and inclusion in the precedence clause follow the `<tools>`/`<env>`
precedent exactly: memory content is model-written from untrusted input and must
be framed as data, never instructions. The rendered block carries each record's
short id, because the write path addresses records by id — one source of truth
for both directions.
Both a per-scope record limit and a byte cap on the rendered block are named
constants. The cap is about size, not latency; a local read needs no timeout.
**Prerequisite:** `setupGateways` runs before `setupMemoryAll`
(`main.go:414` vs `:428`), and `startGatewayRuntime` before `setupMemory`
(`gateway.go:110` vs `:115`), so the engine is nil at every turn-runner
construction site. The ordering is the root-cause fix and nothing in either
memory step reads what those later steps produce.
### 5. The chat write path
The memory tool is built **per turn**, not registered once at boot, because it
must write into a scope derived from the turn's resolved subject:
| action | engine call |
|---|---|
| `create` | `Create` |
| `update` | `Update` (by id) |
| `delete` | `Forget` (by id) |
| `list` | `List` |
**The model chooses the scope kind; it never chooses the ids.** Agent and user
ids come from the turn's resolution and there is no schema field to put anything
else in — even a model that emits another user's id has nowhere to write it. That
is the root-cause form of `agent-system.md:87-90`, matching this system's other
non-waivable boundaries (the model never runs git; gates cannot be waived by the
agent). `global` is not offered at all.
The tool takes a record **id** where the retired provider took a content
substring (`handleReplace`/`handleRemove`). The substring addressing existed
because a markdown file was hand-edited and the model had no stable handle on a
block; with ids rendered into the prompt it disappears, and one addressing scheme
replaces two.
### 6. The curator's write
`sessioncurator.reviewOne` stops writing `Identity: sessionID` and writes
`ScopeAgentUser{AgentID: <session's agent>, UserID: <participant>}`. Relationship
scope is the only defensible home: agent-wide is explicitly unsafe for user facts
and would leak user A's extracted facts to user B through the same agent;
user-wide would pool facts across agents; and a session id cannot survive the
session ending, which is the whole requirement.
This needs data the tree does not persist today: the SQLite `messages` table
gains `sender_id`, and `curator.SessionSummary` / `ConversationMessage` gain the
agent and participant. The participant is derived from the session's own
user-role messages: exactly one distinct non-empty sender writes; zero (a
dashboard or webhook session) or several (a group chat) writes nothing and
records why. Ambiguity fails closed rather than guessing.
The payoff is the point of the exercise: chat-written and curator-derived memory
now occupy the same scopes, so a future turn's prompt contains both, and a
curator's bad note becomes id-addressable and therefore correctable by the agent.
### 7. The content scanner moves to the engine
`internal/memory/scanner.go` and its installation at `bootstrap.go:1045` are what
make `configuration.md`'s "Memory safety scanner" row describe a control that
actually runs. Deleting the package would silently remove prompt-injection
scanning on memory writes. The scanner moves into the engine and is applied in
`Create` and `Update`, which is the single choke point every producer crosses —
so the curator's model-extracted content is covered too, not just the chat tool's.
### 8. `internal/memory` is deleted
Not shimmed, not partially, not behind a flag: `manager.go`, `provider.go`,
`lifecycle.go`, `sync.go`, `scanner.go` (moved), `schema.go`, `backup.go`,
`config.go`, `builtin/provider.go`, and their tests. Its markdown store
(`builtin/store.go`) is the one survivor, relocated to
`internal/infrastructure/memory/builtin/` — `organisation.md`'s "a package owns
its on-disk format end-to-end" makes infrastructure its correct home, and the
engine already parses that format.
Also deleted: `internal/tools/provider/memory`'s manager wrapping (rewired to the
engine), `config.Memory.{Provider,ProviderConfig}` and the
`validateMemory` rejection that exists only to complain about `Provider`
(`validate.go:143-154`), and `webui.MemoryStatus`/`MemoryProviderHandle` — never
implemented in production, so `/api/memory` reports empty forever unless it is
repointed at the engine registry.
Existing data at `<workDir>/memory` is left on disk, unreferenced. No migration.
`<workDir>/memory-engine` is likewise orphaned: its directories were keyed by
`sha256(sessionID)` and no scope key can hash to one.
## What we deliberately do NOT do
- **No cross-scope sharing or grants in this wave.** Promoting a record between
scopes is a future explicit command, and `Update` cannot change scope, so it
cannot happen by accident.
- **No canonical user identity.** `SenderID` is channel-native, so one person on
Telegram and email is two identities. Unifying needs an Identity aggregate that
does not exist (`internal/domain` has no identity package and `IdentityID`
appears nowhere in Go today). The contract does not change when it lands.
- **No ranked retrieval.** `Query.Text` stays, unused by the prompt path.
- **No data migration**, per the decision above.
- **No prompt-memory timeout mechanism.** The read is bounded by construction.
- **No `Metadata` field.** It is write-only today; a caller needing it later
needs one field, not a speculative contract.
## Acceptance criteria
1. A record created through the memory tool in turn 1 appears in turn 2's prompt,
asserted on content the test planted, with its id.
2. The same record is **absent** from a different user's prompt on the same
channel and agent. Isolation asserted end to end, not at the engine boundary.
3. An agent-scope record is visible to both of that agent's users; a global
record is visible to a second agent; an agent-user record is not.
4. A curator pass over a session writes a record a later turn reads, and which
survives that session being deleted.
5. `create` → prompt shows it; `update` → prompt shows new content and
`Revisions` returns the prior content with author and time; `delete` → gone
from prompt and `List`. Round-trip is asserted as `parse(render(r)) == r`,
never against a fixture string.
6. A panicking engine and a failing engine both complete the turn with no memory
block.
7. A channel with no resolvable user renders only global and agent scopes, and
the tool refuses `user`/`agent-user` with a reason. A webhook route path never
becomes a user identity.
8. A write that would exceed the file cap fails loudly rather than silently
dropping content.
9. `internal/memory` is gone, `task check` is green, and no document describes a
manager that no longer exists.
Every criterion asserts behaviour or planted content. None asserts on the text of
a hand-written file, and none pins that a method was called. In particular none
asserts the composition calls setup in a given order: if the ordering were
swapped back, criteria 1 and 4 would fail, and that is the assertion.
## Sequencing
Each step is independently verifiable and lands green on its own.
1. **Persist the participant.** `sender_id` column on `messages`,
insert/select/migration, `curator.SessionSummary.AgentID` and
`ConversationMessage.SenderID`. Touches no memory contract and is shippable
alone — nothing consumes it yet.
2. **Scope, CRUD and revisions.** The `internal/domain/memory` contract, the
builtin engine with `HISTORY.md`, the store relocation, the scanner move, and
the curator rewritten onto agent-user scope (it is the engine's only live
consumer, so it belongs in this slice). *Depends on 1 for the curator half.*
3. **Chat read path.** The `Memory` seam, the template block, subject resolution,
and the composition-ordering fix. *Depends on 2.*
4. **Chat write path.** The tool adapter onto CRUD and the turn's subject on the
context. *Depends on 3, because a write is only observable through the read.*
5. **Delete `internal/memory` and the dead config/dashboard surface**, and
rewrite `ARCHITECTURE.md`'s "Memory engine family" section to describe the
engine family rather than a manager it deleted. *Depends on 3 and 4.*
Steps 3 and 4 are separable from each other; step 1 is separable from everything.
## Settles these open items in `migration-decisions.md` §5
| §5 open item | Settled by |
|---|---|
| authoritative memory record and revision model | §1, §2 — the live marker becomes JSON, since the ratified `<!--mem:<ulid>-->` cannot carry provenance; superseded states live in a per-scope `HISTORY.md` |
| provider and infrastructure boundaries | §2 — contract in `internal/domain/memory`, builtin format in `internal/infrastructure/memory/builtin` |
| retrieval and access enforcement | §1 — caller-named scope sets, no engine-side policy |
| provenance representation | §2 |
## Not determined
- **Group chats produce no derived memory**, because the curator fails closed on
mixed-sender sessions. Whether extraction should instead become
per-participant is a product call.
- **Two processes, one scope file.** After step 3 both `archied` and
`archie-gateway` hold an engine over the same root, and the engine caches a
store per scope and rewrites the whole file, so concurrent writers on one scope
can lose a block. `Update.Expected` makes that detectable for one record; it
does not make the file safe. A reload-before-mutate or advisory lock is the fix
and is not designed here.
- **Unbounded history.** `HISTORY.md` grows without limit; no compaction policy.
- **Size bounds.** `defaultMaxFileBytes` is now per scope rather than per
identity; whether that is right is unexamined.
- **Whether the dashboard should carry a user identity.** It has one bearer
token, not users. Until it has users, web turns are correctly limited to global
and agent scope.