docs(prds): land the memory engine unification decision
Approved for implementation, dated 2026-09-16, previously untracked.
Settles four open items in docs/architecture/migration-decisions.md#5 and
supersedes the memory-turn-context.md draft.
Section 8's deletion list no longer names config.Memory.SessionTTL: the
council carried that deletion standalone and f25dd1e removed the field,
so listing it here would schedule work already done. Provider and
ProviderConfig are unchanged, both still present.
This commit is contained in:
1 file changed
+362
@@ -0,0 +1,362 @@
|
||||
# Memory engine unification: four scopes, CRUD, and one write path -- decision
|
||||
|
||||
**Status:** Approved for implementation
|
||||
**Date:** 2026-09-16
|
||||
**Findings:** `memory-4` and its dependants in the adversarial sweep
|
||||
(`.local/issue-tracker`); the real defect is the split described below, for
|
||||
which `memory-4` is a symptom
|
||||
**Supersedes:** the draft `memory-turn-context.md` (its synchronous-read
|
||||
argument survives and is reused; its `Manager` method and consumer-owned port do
|
||||
not)
|
||||
**Settles:** four open items in `docs/architecture/migration-decisions.md#5`
|
||||
(listed at the end)
|
||||
|
||||
## Problem
|
||||
|
||||
There are **two live memory stores for one assistant**, split by which component
|
||||
writes them, and the chat turn reads neither.
|
||||
|
||||
| Family | Written by | Root | Format |
|
||||
|---|---|---|---|
|
||||
| `internal/memory` (`Manager`, `builtin.Provider`) | the chat agent's `memory_edit` tool via `memorytoolprovider` (`bootstrap.go:1198`) | `<workDir>/memory` | `MEMORY.md` / `USER.md` |
|
||||
| `internal/domain/memory` + `internal/infrastructure/memory` | the session curator (`sessioncurator.go:163`) | `<workDir>/memory-engine` | identity-keyed blocks |
|
||||
|
||||
They share no code, no format and no scope vocabulary. Neither reaches a prompt:
|
||||
`Manager.SystemPromptBlock` (`internal/memory/manager.go:429`) has no production
|
||||
caller anywhere, so the agent can write memory through a tool and never read it
|
||||
back.
|
||||
|
||||
Three further defects fall out of the same split:
|
||||
|
||||
1. **The engine's contract is not CRUD.** `Store` is `Write`/`Query`/`List`/
|
||||
`Forget` — create, read, delete, and **no Update**. The retired file provider
|
||||
*does* have `handleAdd`/`handleReplace`/`handleRemove`, so the family being
|
||||
deleted is operationally richer than its replacement.
|
||||
2. **There is no scope model.** The contract carries one opaque `Identity
|
||||
string`, and the curator fills it with a **session id** (`sessioncurator.go:164`)
|
||||
— which is none of the four scopes `docs/architecture/agent-system.md:69-78`
|
||||
requires, and stops meaning anything when the session ends.
|
||||
3. **The chat turn has no user identity on half the channels.** `SenderID` is
|
||||
populated by Telegram and email (a person) and by webhook (a **route path**,
|
||||
`webhook.go:179`), and not at all by the dashboard, whose gateway wire message
|
||||
carries no sender field. It is also never persisted: `sender_id` appears
|
||||
nowhere in `session_store_sqlite.go`.
|
||||
|
||||
`agent-system.md:100` says the write-scope selection and cross-scope sharing
|
||||
rules "require a focused memory design decision". This is that decision.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. One engine, addressed by four typed scopes
|
||||
|
||||
`internal/domain/memory` is the only memory engine. `Observation.Identity`,
|
||||
`Query.Identity`, `Record.Identity` and `List(identity)` are deleted, not
|
||||
extended: a field whose meaning depends on the caller cannot carry a scope.
|
||||
|
||||
```go
|
||||
type ScopeKind string
|
||||
|
||||
const (
|
||||
ScopeGlobal ScopeKind = "global" // shared across the application
|
||||
ScopeAgent ScopeKind = "agent" // private to one Agent
|
||||
ScopeUser ScopeKind = "user" // one user, across their Agents
|
||||
ScopeAgentUser ScopeKind = "agent-user" // one Agent's relationship with one user
|
||||
)
|
||||
|
||||
type Scope struct {
|
||||
Kind ScopeKind
|
||||
Agent AgentID // required for ScopeAgent, ScopeAgentUser
|
||||
User IdentityID // required for ScopeUser, ScopeAgentUser
|
||||
}
|
||||
|
||||
func (s Scope) Validate() error // rejects a component present where it is not required
|
||||
func (s Scope) Key() string // canonical, collision-free storage key
|
||||
```
|
||||
|
||||
`AgentID` and `IdentityID` are opaque. This package never resolves or
|
||||
canonicalises them.
|
||||
|
||||
A turn's resolved identity is one value:
|
||||
|
||||
```go
|
||||
type Subject struct {
|
||||
AgentID AgentID
|
||||
UserID IdentityID // "" when the channel carries no per-person identity
|
||||
}
|
||||
|
||||
func (s Subject) Scopes() []Scope // readable, most specific first
|
||||
func (s Subject) WritableScopes() []Scope // never includes ScopeGlobal
|
||||
```
|
||||
|
||||
**Access control is naming a scope, and nothing else.** The read path names
|
||||
exactly the scopes of the resolved `Subject`; the engine never unions beyond what
|
||||
it is handed and applies no policy of its own. An engine that enforced policy
|
||||
would need to know about channels, sessions and bindings, and every backend would
|
||||
reimplement it. "Agent-wide is private to its owning Agent by default"
|
||||
(`agent-system.md:80-81`) falls out mechanically, because `ScopeAgent{a}` and
|
||||
`ScopeAgent{b}` are different storage keys.
|
||||
|
||||
`ScopeGlobal` is never model-writable. Writing it is an operator action; nothing
|
||||
produces one yet, but the contract models it because the read set includes it.
|
||||
|
||||
### 2. CRUD, with revisions retained
|
||||
|
||||
```go
|
||||
type Store interface {
|
||||
Create(ctx context.Context, in NewRecord) (Record, error)
|
||||
Get(ctx context.Context, scope Scope, id RecordID) (Record, error) // ErrNotFound
|
||||
Query(ctx context.Context, q Query) ([]Record, error) // heads only
|
||||
List(ctx context.Context, scope Scope) ([]Record, error) // heads only
|
||||
Update(ctx context.Context, in RecordUpdate) (Record, error) // ErrNotFound, ErrStaleRevision
|
||||
Forget(ctx context.Context, scope Scope, id RecordID) error // idempotent
|
||||
Revisions(ctx context.Context, scope Scope, id RecordID) ([]Revision, error)
|
||||
}
|
||||
```
|
||||
|
||||
`Get` and `Forget` take a scope. The current `Forget` scans every identity's
|
||||
store to find a marker (`builtin_engine.go:189-210`) purely because the contract
|
||||
gave it nothing better.
|
||||
|
||||
`Record` carries what `agent-system.md:97-99` requires be retained: `Scope`,
|
||||
`Kind`, `Content`, `Revision`, `Author`, `OriginUser`, `Source`, `CreatedAt`,
|
||||
`UpdatedAt`. `Metadata` is dropped — nothing sets it and the parser never
|
||||
restores it, so it is currently a field that silently does not round-trip.
|
||||
|
||||
`Update` supersedes rather than overwrites. The superseded state is appended to
|
||||
the scope's `HISTORY.md` **before** the live block is replaced, so a crash
|
||||
between the two leaves an extra history entry and an unchanged record —
|
||||
recoverable — never a record whose previous state was lost. `Update` carries no
|
||||
`Scope`: a record may never move between scopes, and the API makes that
|
||||
unrepresentable rather than validating it. `Expected` optionally names the
|
||||
revision being replaced and returns `ErrStaleRevision` on mismatch, which matters
|
||||
because two processes (`archied` and `archie-gateway`) can hold the same scope's
|
||||
file.
|
||||
|
||||
`Forget` records a deleted revision before removing the live block, so a record's
|
||||
provenance outlives its content.
|
||||
|
||||
### 3. The caller resolves identity; the engine never does
|
||||
|
||||
Resolution happens in the gateway turn path, where the native sender id and the
|
||||
session are both in hand, and it is supplied **per channel by the composition**:
|
||||
|
||||
```go
|
||||
// TurnRunnerConfig
|
||||
// UserIdentity resolves the initiating user's identity for one inbound message.
|
||||
// false means the channel carries no per-person identity, and the turn gets
|
||||
// global and agent scopes only. Nil means the same.
|
||||
UserIdentity func(msg messaging.Message) (IdentityID, bool)
|
||||
```
|
||||
|
||||
This must be injectable rather than a blanket `channel + ":" + SenderID`
|
||||
concatenation, because a **webhook's `SenderID` is a route path**
|
||||
(`webhook.go:179`), not a person; treating it as one would give a URL an
|
||||
identity. Telegram and email supply their sender id; the dashboard and webhook
|
||||
resolve nothing.
|
||||
|
||||
**Fail closed.** When `UserID` is empty the turn proceeds with global and agent
|
||||
scopes only, and a write targeting `user` or `agent-user` fails with a clear
|
||||
error. `agent-system.md:92-94` forbids showing one user's memory to another, so
|
||||
"we do not know who this is" must mean "show none of it", never "fall back to a
|
||||
wider scope".
|
||||
|
||||
### 4. The chat read path
|
||||
|
||||
Synchronous and total. It runs inside `prepareTurn` (`turn.go:287`) before the
|
||||
prompt is built, never returns an error to the caller, and degrades to an empty
|
||||
block on any engine failure or provider panic. A fire-and-forget hook would race
|
||||
the prompt build in the same turn, which is why this is not a lifecycle hook.
|
||||
|
||||
The read is a scope-only `Query` over the subject's scopes — **not** ranked
|
||||
search, which remains open in `migration-decisions.md#5`. A ranked read on the
|
||||
hot path with no ranking implementation would be a lie.
|
||||
|
||||
`SystemPromptConfig` gains a `Memory` field, rendered into
|
||||
`templates/archie.md.tpl` between `<tools>` (line 41) and `<env>` (line 50):
|
||||
|
||||
```
|
||||
<memory purpose="durable_context" trust="data">
|
||||
```
|
||||
|
||||
`trust="data"` and inclusion in the precedence clause follow the `<tools>`/`<env>`
|
||||
precedent exactly: memory content is model-written from untrusted input and must
|
||||
be framed as data, never instructions. The rendered block carries each record's
|
||||
short id, because the write path addresses records by id — one source of truth
|
||||
for both directions.
|
||||
|
||||
Both a per-scope record limit and a byte cap on the rendered block are named
|
||||
constants. The cap is about size, not latency; a local read needs no timeout.
|
||||
|
||||
**Prerequisite:** `setupGateways` runs before `setupMemoryAll`
|
||||
(`main.go:414` vs `:428`), and `startGatewayRuntime` before `setupMemory`
|
||||
(`gateway.go:110` vs `:115`), so the engine is nil at every turn-runner
|
||||
construction site. The ordering is the root-cause fix and nothing in either
|
||||
memory step reads what those later steps produce.
|
||||
|
||||
### 5. The chat write path
|
||||
|
||||
The memory tool is built **per turn**, not registered once at boot, because it
|
||||
must write into a scope derived from the turn's resolved subject:
|
||||
|
||||
| action | engine call |
|
||||
|---|---|
|
||||
| `create` | `Create` |
|
||||
| `update` | `Update` (by id) |
|
||||
| `delete` | `Forget` (by id) |
|
||||
| `list` | `List` |
|
||||
|
||||
**The model chooses the scope kind; it never chooses the ids.** Agent and user
|
||||
ids come from the turn's resolution and there is no schema field to put anything
|
||||
else in — even a model that emits another user's id has nowhere to write it. That
|
||||
is the root-cause form of `agent-system.md:87-90`, matching this system's other
|
||||
non-waivable boundaries (the model never runs git; gates cannot be waived by the
|
||||
agent). `global` is not offered at all.
|
||||
|
||||
The tool takes a record **id** where the retired provider took a content
|
||||
substring (`handleReplace`/`handleRemove`). The substring addressing existed
|
||||
because a markdown file was hand-edited and the model had no stable handle on a
|
||||
block; with ids rendered into the prompt it disappears, and one addressing scheme
|
||||
replaces two.
|
||||
|
||||
### 6. The curator's write
|
||||
|
||||
`sessioncurator.reviewOne` stops writing `Identity: sessionID` and writes
|
||||
`ScopeAgentUser{AgentID: <session's agent>, UserID: <participant>}`. Relationship
|
||||
scope is the only defensible home: agent-wide is explicitly unsafe for user facts
|
||||
and would leak user A's extracted facts to user B through the same agent;
|
||||
user-wide would pool facts across agents; and a session id cannot survive the
|
||||
session ending, which is the whole requirement.
|
||||
|
||||
This needs data the tree does not persist today: the SQLite `messages` table
|
||||
gains `sender_id`, and `curator.SessionSummary` / `ConversationMessage` gain the
|
||||
agent and participant. The participant is derived from the session's own
|
||||
user-role messages: exactly one distinct non-empty sender writes; zero (a
|
||||
dashboard or webhook session) or several (a group chat) writes nothing and
|
||||
records why. Ambiguity fails closed rather than guessing.
|
||||
|
||||
The payoff is the point of the exercise: chat-written and curator-derived memory
|
||||
now occupy the same scopes, so a future turn's prompt contains both, and a
|
||||
curator's bad note becomes id-addressable and therefore correctable by the agent.
|
||||
|
||||
### 7. The content scanner moves to the engine
|
||||
|
||||
`internal/memory/scanner.go` and its installation at `bootstrap.go:1045` are what
|
||||
make `configuration.md`'s "Memory safety scanner" row describe a control that
|
||||
actually runs. Deleting the package would silently remove prompt-injection
|
||||
scanning on memory writes. The scanner moves into the engine and is applied in
|
||||
`Create` and `Update`, which is the single choke point every producer crosses —
|
||||
so the curator's model-extracted content is covered too, not just the chat tool's.
|
||||
|
||||
### 8. `internal/memory` is deleted
|
||||
|
||||
Not shimmed, not partially, not behind a flag: `manager.go`, `provider.go`,
|
||||
`lifecycle.go`, `sync.go`, `scanner.go` (moved), `schema.go`, `backup.go`,
|
||||
`config.go`, `builtin/provider.go`, and their tests. Its markdown store
|
||||
(`builtin/store.go`) is the one survivor, relocated to
|
||||
`internal/infrastructure/memory/builtin/` — `organisation.md`'s "a package owns
|
||||
its on-disk format end-to-end" makes infrastructure its correct home, and the
|
||||
engine already parses that format.
|
||||
|
||||
Also deleted: `internal/tools/provider/memory`'s manager wrapping (rewired to the
|
||||
engine), `config.Memory.{Provider,ProviderConfig}` and the
|
||||
`validateMemory` rejection that exists only to complain about `Provider`
|
||||
(`validate.go:143-154`), and `webui.MemoryStatus`/`MemoryProviderHandle` — never
|
||||
implemented in production, so `/api/memory` reports empty forever unless it is
|
||||
repointed at the engine registry.
|
||||
|
||||
Existing data at `<workDir>/memory` is left on disk, unreferenced. No migration.
|
||||
`<workDir>/memory-engine` is likewise orphaned: its directories were keyed by
|
||||
`sha256(sessionID)` and no scope key can hash to one.
|
||||
|
||||
## What we deliberately do NOT do
|
||||
|
||||
- **No cross-scope sharing or grants in this wave.** Promoting a record between
|
||||
scopes is a future explicit command, and `Update` cannot change scope, so it
|
||||
cannot happen by accident.
|
||||
- **No canonical user identity.** `SenderID` is channel-native, so one person on
|
||||
Telegram and email is two identities. Unifying needs an Identity aggregate that
|
||||
does not exist (`internal/domain` has no identity package and `IdentityID`
|
||||
appears nowhere in Go today). The contract does not change when it lands.
|
||||
- **No ranked retrieval.** `Query.Text` stays, unused by the prompt path.
|
||||
- **No data migration**, per the decision above.
|
||||
- **No prompt-memory timeout mechanism.** The read is bounded by construction.
|
||||
- **No `Metadata` field.** It is write-only today; a caller needing it later
|
||||
needs one field, not a speculative contract.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
1. A record created through the memory tool in turn 1 appears in turn 2's prompt,
|
||||
asserted on content the test planted, with its id.
|
||||
2. The same record is **absent** from a different user's prompt on the same
|
||||
channel and agent. Isolation asserted end to end, not at the engine boundary.
|
||||
3. An agent-scope record is visible to both of that agent's users; a global
|
||||
record is visible to a second agent; an agent-user record is not.
|
||||
4. A curator pass over a session writes a record a later turn reads, and which
|
||||
survives that session being deleted.
|
||||
5. `create` → prompt shows it; `update` → prompt shows new content and
|
||||
`Revisions` returns the prior content with author and time; `delete` → gone
|
||||
from prompt and `List`. Round-trip is asserted as `parse(render(r)) == r`,
|
||||
never against a fixture string.
|
||||
6. A panicking engine and a failing engine both complete the turn with no memory
|
||||
block.
|
||||
7. A channel with no resolvable user renders only global and agent scopes, and
|
||||
the tool refuses `user`/`agent-user` with a reason. A webhook route path never
|
||||
becomes a user identity.
|
||||
8. A write that would exceed the file cap fails loudly rather than silently
|
||||
dropping content.
|
||||
9. `internal/memory` is gone, `task check` is green, and no document describes a
|
||||
manager that no longer exists.
|
||||
|
||||
Every criterion asserts behaviour or planted content. None asserts on the text of
|
||||
a hand-written file, and none pins that a method was called. In particular none
|
||||
asserts the composition calls setup in a given order: if the ordering were
|
||||
swapped back, criteria 1 and 4 would fail, and that is the assertion.
|
||||
|
||||
## Sequencing
|
||||
|
||||
Each step is independently verifiable and lands green on its own.
|
||||
|
||||
1. **Persist the participant.** `sender_id` column on `messages`,
|
||||
insert/select/migration, `curator.SessionSummary.AgentID` and
|
||||
`ConversationMessage.SenderID`. Touches no memory contract and is shippable
|
||||
alone — nothing consumes it yet.
|
||||
2. **Scope, CRUD and revisions.** The `internal/domain/memory` contract, the
|
||||
builtin engine with `HISTORY.md`, the store relocation, the scanner move, and
|
||||
the curator rewritten onto agent-user scope (it is the engine's only live
|
||||
consumer, so it belongs in this slice). *Depends on 1 for the curator half.*
|
||||
3. **Chat read path.** The `Memory` seam, the template block, subject resolution,
|
||||
and the composition-ordering fix. *Depends on 2.*
|
||||
4. **Chat write path.** The tool adapter onto CRUD and the turn's subject on the
|
||||
context. *Depends on 3, because a write is only observable through the read.*
|
||||
5. **Delete `internal/memory` and the dead config/dashboard surface**, and
|
||||
rewrite `ARCHITECTURE.md`'s "Memory engine family" section to describe the
|
||||
engine family rather than a manager it deleted. *Depends on 3 and 4.*
|
||||
|
||||
Steps 3 and 4 are separable from each other; step 1 is separable from everything.
|
||||
|
||||
## Settles these open items in `migration-decisions.md` §5
|
||||
|
||||
| §5 open item | Settled by |
|
||||
|---|---|
|
||||
| authoritative memory record and revision model | §1, §2 — the live marker becomes JSON, since the ratified `<!--mem:<ulid>-->` cannot carry provenance; superseded states live in a per-scope `HISTORY.md` |
|
||||
| provider and infrastructure boundaries | §2 — contract in `internal/domain/memory`, builtin format in `internal/infrastructure/memory/builtin` |
|
||||
| retrieval and access enforcement | §1 — caller-named scope sets, no engine-side policy |
|
||||
| provenance representation | §2 |
|
||||
|
||||
## Not determined
|
||||
|
||||
- **Group chats produce no derived memory**, because the curator fails closed on
|
||||
mixed-sender sessions. Whether extraction should instead become
|
||||
per-participant is a product call.
|
||||
- **Two processes, one scope file.** After step 3 both `archied` and
|
||||
`archie-gateway` hold an engine over the same root, and the engine caches a
|
||||
store per scope and rewrites the whole file, so concurrent writers on one scope
|
||||
can lose a block. `Update.Expected` makes that detectable for one record; it
|
||||
does not make the file safe. A reload-before-mutate or advisory lock is the fix
|
||||
and is not designed here.
|
||||
- **Unbounded history.** `HISTORY.md` grows without limit; no compaction policy.
|
||||
- **Size bounds.** `defaultMaxFileBytes` is now per scope rather than per
|
||||
identity; whether that is right is unexamined.
|
||||
- **Whether the dashboard should carry a user identity.** It has one bearer
|
||||
token, not users. Until it has users, web turns are correctly limited to global
|
||||
and agent scope.
|
||||
Reference in new issue
Block a user