ARCHITECTURE.md duplicated the docs/architecture tree and had drifted badly
against it: "internal/ holds ~36" against 112 packages, two of the seven
binaries listed, a Forge table listing `RepliesAfter` (deleted from the tree),
and a stated invariant ("No daemon-internal cron") contradicted by the
scheduling engine that internal/app/archied/scheduling.go composes and
main.go starts.
It was also the sole home of three pieces of still-authoritative content, so
those move rather than disappear:
- plugin engine rule (strict), the memory engine family, and the
internal/plugin/architecture_test.go enforcement note ->
docs/architecture/plugins-and-extensions.md
- intake provenance: forge poll/webhook dedup through
TaskEnvelope.IdempotencyKey, binding-stamped chat/playbook enqueue, and the
single-identity webhook-intake rejection ->
docs/architecture/messaging-and-work-intake.md
Live references are repointed from CLAUDE.md (AGENTS.md is a symlink to it),
docs/architecture, docs/guides, seven PRDs, and four .agents skills. The
heading "## Plugin engine rule (strict)" is preserved in its new home so the
existing #plugin-engine-rule-strict citations keep resolving. Generated docs
artifacts regenerated.
18 KiB
Memory engine unification: four scopes, CRUD, and one write path -- decision
Status: Approved
Date: 2026-09-16
Findings: memory-4 and its dependants in the adversarial sweep
(.local/issue-tracker); the real defect is the split described below, for
which memory-4 is a symptom
Supersedes: the draft memory-turn-context.md (its synchronous-read
argument survives and is reused; its Manager method and consumer-owned port do
not)
Settles: four open items in docs/architecture/migration-decisions.md#5
(listed at the end)
Problem
There are two live memory stores for one assistant, split by which component writes them, and the chat turn reads neither.
| Family | Written by | Root | Format |
|---|---|---|---|
internal/memory (Manager, builtin.Provider) |
the chat agent's memory_edit tool via memorytoolprovider (bootstrap.go) |
<workDir>/memory |
MEMORY.md / USER.md |
internal/domain/memory + internal/infrastructure/memory |
the session curator (sessioncurator.go) |
<workDir>/memory-engine |
identity-keyed blocks |
They share no code, no format and no scope vocabulary. Neither reaches a prompt:
Manager.SystemPromptBlock (internal/memory/manager.go) has no production
caller anywhere, so the agent can write memory through a tool and never read it
back.
Three further defects fall out of the same split:
- The engine's contract is not CRUD.
StoreisWrite/Query/List/Forget— create, read, delete, and no Update. The retired file provider does havehandleAdd/handleReplace/handleRemove, so the family being deleted is operationally richer than its replacement. - There is no scope model. The contract carries one opaque
Identity string, and the curator fills it with a session id (sessioncurator.go) — which is none of the four scopesdocs/architecture/agent-system.mdrequires, and stops meaning anything when the session ends. - The chat turn has no user identity on half the channels. Telegram and
email populate
SenderIDwith a person; the dashboard populates it with nothing, and a webhook populates it with nothing either — a route path names a source, not a person, sowebhook.goleavesSenderIDempty and carries the route inInbound.BudgetKeyfor rate limiting.sender_idis persisted (session_store_sqlite.go), so whatever a channel puts there reaches the consumers that read it.
agent-system.md says the write-scope selection and cross-scope sharing
rules "require a focused memory design decision". This is that decision.
Decision
1. One engine, addressed by four typed scopes
internal/domain/memory is the only memory engine. Observation.Identity,
Query.Identity, Record.Identity and List(identity) are deleted, not
extended: a field whose meaning depends on the caller cannot carry a scope.
type ScopeKind string
const (
ScopeGlobal ScopeKind = "global" // shared across the application
ScopeAgent ScopeKind = "agent" // private to one Agent
ScopeUser ScopeKind = "user" // one user, across their Agents
ScopeAgentUser ScopeKind = "agent-user" // one Agent's relationship with one user
)
type Scope struct {
Kind ScopeKind
Agent AgentID // required for ScopeAgent, ScopeAgentUser
User IdentityID // required for ScopeUser, ScopeAgentUser
}
func (s Scope) Validate() error // rejects a component present where it is not required
func (s Scope) Key() string // canonical, collision-free storage key
AgentID and IdentityID are opaque. This package never resolves or
canonicalises them.
A turn's resolved identity is one value:
type Subject struct {
AgentID AgentID
UserID IdentityID // "" when the channel carries no per-person identity
}
func (s Subject) Scopes() []Scope // readable, most specific first
func (s Subject) WritableScopes() []Scope // never includes ScopeGlobal
Access control is naming a scope, and nothing else. The read path names
exactly the scopes of the resolved Subject; the engine never unions beyond what
it is handed and applies no policy of its own. An engine that enforced policy
would need to know about channels, sessions and bindings, and every backend would
reimplement it. "Agent-wide is private to its owning Agent by default"
(agent-system.md) falls out mechanically, because ScopeAgent{a} and
ScopeAgent{b} are different storage keys.
ScopeGlobal is never model-writable. Writing it is an operator action; nothing
produces one yet, but the contract models it because the read set includes it.
2. CRUD, with revisions retained
type Store interface {
Create(ctx context.Context, in NewRecord) (Record, error)
Get(ctx context.Context, scope Scope, id RecordID) (Record, error) // ErrNotFound
Query(ctx context.Context, q Query) ([]Record, error) // heads only
List(ctx context.Context, scope Scope) ([]Record, error) // heads only
Update(ctx context.Context, in RecordUpdate) (Record, error) // ErrNotFound, ErrStaleRevision
Forget(ctx context.Context, scope Scope, id RecordID) error // idempotent
Revisions(ctx context.Context, scope Scope, id RecordID) ([]Revision, error)
}
Get and Forget take a scope. The current Forget scans every identity's
store to find a marker (builtin_engine.go) purely because the contract
gave it nothing better.
Record carries what agent-system.md requires be retained: Scope,
Kind, Content, Revision, Author, OriginUser, Source, CreatedAt,
UpdatedAt. Metadata is dropped — nothing sets it and the parser never
restores it, so it is a field that silently does not round-trip.
Update supersedes rather than overwrites. The superseded state is appended to
the scope's HISTORY.md before the live block is replaced, so a crash
between the two leaves an extra history entry and an unchanged record —
recoverable — never a record whose previous state was lost. Update carries no
Scope: a record may never move between scopes, and the API makes that
unrepresentable rather than validating it. Expected optionally names the
revision being replaced and returns ErrStaleRevision on mismatch, which matters
because two processes (archied and archie-gateway) can hold the same scope's
file.
Forget records a deleted revision before removing the live block, so a record's
provenance outlives its content.
3. The caller resolves identity; the engine never does
Resolution happens in the gateway turn path, where the native sender id and the session are both in hand, and it is supplied per channel by the composition:
// TurnRunnerConfig
// UserIdentity resolves the initiating user's identity for one inbound message.
// false means the channel carries no per-person identity, and the turn gets
// global and agent scopes only. Nil means the same.
UserIdentity func(msg messaging.Message) (IdentityID, bool)
This must be injectable rather than a blanket channel + ":" + SenderID
concatenation, because a channel's sender id is not necessarily a person: a
webhook's route path names a source, and treating it as an identity would give
a URL one -- which is why webhook.go leaves SenderID empty and carries the
route in Inbound.BudgetKey instead. Telegram and email supply their sender
id; the dashboard and webhook resolve nothing.
Fail closed. When UserID is empty the turn proceeds with global and agent
scopes only, and a write targeting user or agent-user fails with a clear
error. agent-system.md forbids showing one user's memory to another, so
"we do not know who this is" must mean "show none of it", never "fall back to a
wider scope".
4. The chat read path
Synchronous and total. It runs inside prepareTurn (turn.go) before the
prompt is built, never returns an error to the caller, and degrades to an empty
block on any engine failure or provider panic. A fire-and-forget hook would race
the prompt build in the same turn, which is why this is not a lifecycle hook.
The read is a scope-only Query over the subject's scopes — not ranked
search, which remains open in migration-decisions.md#5. A ranked read on the
hot path with no ranking implementation would be a lie.
SystemPromptConfig gains a Memory field, rendered into
templates/archie.md.tpl between the <tools> and <env> blocks:
<memory purpose="durable_context" trust="data">
trust="data" and inclusion in the precedence clause follow the <tools>/<env>
precedent exactly: memory content is model-written from untrusted input and must
be framed as data, never instructions. The rendered block carries each record's
short id, because the write path addresses records by id — one source of truth
for both directions.
Both a per-scope record limit and a byte cap on the rendered block are named constants. The cap is about size, not latency; a local read needs no timeout.
Prerequisite: setupGateways runs before setupMemoryAll
(main.go vs :428), and startGatewayRuntime before setupMemory
(gateway.go vs :115), so the engine is nil at every turn-runner
construction site. The ordering is the root-cause fix and nothing in either
memory step reads what those later steps produce.
5. The chat write path
The memory tool is built per turn, not registered once at boot, because it must write into a scope derived from the turn's resolved subject:
| action | engine call |
|---|---|
create |
Create |
update |
Update (by id) |
delete |
Forget (by id) |
list |
List |
The model chooses the scope kind; it never chooses the ids. Agent and user
ids come from the turn's resolution and there is no schema field to put anything
else in — even a model that emits another user's id has nowhere to write it. That
is the root-cause form of agent-system.md, matching this system's other
non-waivable boundaries (the model never runs git; gates cannot be waived by the
agent). global is not offered at all.
The tool takes a record id where the retired provider took a content
substring (handleReplace/handleRemove). The substring addressing existed
because a markdown file was hand-edited and the model had no stable handle on a
block; with ids rendered into the prompt it disappears, and one addressing scheme
replaces two.
6. The curator's write
sessioncurator.reviewOne stops writing Identity: sessionID and writes
ScopeAgentUser{AgentID: <session's agent>, UserID: <participant>}. Relationship
scope is the only defensible home: agent-wide is explicitly unsafe for user facts
and would leak user A's extracted facts to user B through the same agent;
user-wide would pool facts across agents; and a session id cannot survive the
session ending, which is the whole requirement.
This needs data the tree does not persist: the SQLite messages table
gains sender_id, and curator.SessionSummary / ConversationMessage gain the
agent and participant. The participant is derived from the session's own
user-role messages: exactly one distinct non-empty sender writes; zero (a
dashboard or webhook session) or several (a group chat) writes nothing and
records why. Ambiguity fails closed rather than guessing.
The payoff is the point of the exercise: chat-written and curator-derived memory now occupy the same scopes, so a future turn's prompt contains both, and a curator's bad note becomes id-addressable and therefore correctable by the agent.
7. The content scanner moves to the engine
internal/memory/scanner.go and its installation at bootstrap.go are what
make configuration.md's "Memory safety scanner" row describe a control that
actually runs. Deleting the package would silently remove prompt-injection
scanning on memory writes. The scanner moves into the engine and is applied in
Create and Update, which is the single choke point every producer crosses —
so the curator's model-extracted content is covered too, not just the chat tool's.
8. internal/memory is deleted
Not shimmed, not partially, not behind a flag: manager.go, provider.go,
lifecycle.go, sync.go, scanner.go (moved), schema.go, backup.go,
config.go, builtin/provider.go, and their tests. Its markdown store
(builtin/store.go) is the one survivor, relocated to
internal/infrastructure/memory/builtin/ — organisation.md's "a package owns
its on-disk format end-to-end" makes infrastructure its correct home, and the
engine already parses that format.
Also deleted: internal/tools/provider/memory's manager wrapping (rewired to the
engine), config.Memory.{Provider,ProviderConfig} and the
validateMemory rejection that exists only to complain about Provider
(validate.go), and webui.MemoryStatus/MemoryProviderHandle — never
implemented in production, so /api/memory reports empty forever unless it is
repointed at the engine registry.
Existing data at <workDir>/memory is left on disk, unreferenced. No migration.
<workDir>/memory-engine is likewise orphaned: its directories were keyed by
sha256(sessionID) and no scope key can hash to one.
What we deliberately do NOT do
- No cross-scope sharing or grants in this wave. Promoting a record between
scopes is a future explicit command, and
Updatecannot change scope, so it cannot happen by accident. - No canonical user identity.
SenderIDis channel-native, so one person on Telegram and email is two identities. Unifying needs an Identity aggregate that does not exist (internal/domainhas no identity package andIdentityIDappears nowhere in Go). The contract does not change when it lands. - No ranked retrieval.
Query.Textstays, unused by the prompt path. - No data migration, per the decision above.
- No prompt-memory timeout mechanism. The read is bounded by construction.
- No
Metadatafield. It is write-only; a caller needing it later needs one field, not a speculative contract.
Acceptance criteria
- A record created through the memory tool in turn 1 appears in turn 2's prompt, asserted on content the test planted, with its id.
- The same record is absent from a different user's prompt on the same channel and agent. Isolation asserted end to end, not at the engine boundary.
- An agent-scope record is visible to both of that agent's users; a global record is visible to a second agent; an agent-user record is not.
- A curator pass over a session writes a record a later turn reads, and which survives that session being deleted.
create→ prompt shows it;update→ prompt shows new content andRevisionsreturns the prior content with author and time;delete→ gone from prompt andList. Round-trip is asserted asparse(render(r)) == r, never against a fixture string.- A panicking engine and a failing engine both complete the turn with no memory block.
- A channel with no resolvable user renders only global and agent scopes, and
the tool refuses
user/agent-userwith a reason. A webhook route path never becomes a user identity. - A write that would exceed the file cap fails loudly rather than silently dropping content.
internal/memoryis gone,task checkis green, and no document describes a manager that no longer exists.
Every criterion asserts behaviour or planted content. None asserts on the text of a hand-written file, and none pins that a method was called. In particular none asserts the composition calls setup in a given order: if the ordering were swapped back, criteria 1 and 4 would fail, and that is the assertion.
Sequencing
Each step is independently verifiable and lands green on its own.
- Persist the participant.
sender_idcolumn onmessages, insert/select/migration,curator.SessionSummary.AgentIDandConversationMessage.SenderID. Touches no memory contract and is shippable alone — nothing consumes it yet. - Scope, CRUD and revisions. The
internal/domain/memorycontract, the builtin engine withHISTORY.md, the store relocation, the scanner move, and the curator rewritten onto agent-user scope (it is the engine's only live consumer, so it belongs in this slice). Depends on 1 for the curator half. - Chat read path. The
Memoryseam, the template block, subject resolution, and the composition-ordering fix. Depends on 2. - Chat write path. The tool adapter onto CRUD and the turn's subject on the context. Depends on 3, because a write is only observable through the read.
- Delete
internal/memoryand the dead config/dashboard surface, and rewrite the "Memory engine family" section ofdocs/architecture/plugins-and-extensions.mdto describe the engine family rather than a manager it deleted. Depends on 3 and 4.
Steps 3 and 4 are separable from each other; step 1 is separable from everything.
Settles these open items in migration-decisions.md §5
| §5 open item | Settled by |
|---|---|
| authoritative memory record and revision model | §1, §2 — the live marker becomes JSON, since the ratified <!--mem:<ulid>--> cannot carry provenance; superseded states live in a per-scope HISTORY.md |
| provider and infrastructure boundaries | §2 — contract in internal/domain/memory, builtin format in internal/infrastructure/memory/builtin |
| retrieval and access enforcement | §1 — caller-named scope sets, no engine-side policy |
| provenance representation | §2 |
Not determined
- Group chats produce no derived memory, because the curator fails closed on mixed-sender sessions. Whether extraction should instead become per-participant is a product call.
- Two processes, one scope file. After step 3 both
archiedandarchie-gatewayhold an engine over the same root, and the engine caches a store per scope and rewrites the whole file, so concurrent writers on one scope can lose a block.Update.Expectedmakes that detectable for one record; it does not make the file safe. A reload-before-mutate or advisory lock is the fix and is not designed here. - Unbounded history.
HISTORY.mdgrows without limit; no compaction policy. - Size bounds.
defaultMaxFileBytesis now per scope rather than per identity; whether that is right is unexamined. - Whether the dashboard should carry a user identity. It has one bearer token, not users. Until it has users, web turns are correctly limited to global and agent scope.