feat(memory): four-scope contract and builtin engine CRUD (unification slice 2) (#886)

* feat(memory): four-scope contract with CRUD and revisions

Seam for memory unification slice 2 (docs/prds/memory-engine-unification.md
§1, §2, §7): internal/domain/memory is now a scope-addressed CRUD contract.

- Scope{Kind,Agent,User} with the four ratified kinds, Validate rejecting a
  component present where it is not required, and a length-prefixed Key so
  opaque channel-native ids cannot collide.
- Subject{AgentID,UserID} with Scopes()/WritableScopes(): global is never
  writable, and an absent user id removes every scope requiring one.
- Store becomes Create/Get/Query/List/Update/Forget/Revisions over
  NewRecord/RecordUpdate/Record/Revision, with ErrNotFound/ErrStaleRevision.
  Observation, Identity and Metadata are deleted.
- The content scanner moves into the family (internal/domain/memory) so every
  engine applies it in Create and Update; internal/memory re-exports the types
  until slice 5 deletes it.
- The markdown store moves to internal/infrastructure/memory/builtin, its
  correct home per organisation.md.

WIP: internal/infrastructure/memory and internal/infrastructure/sessioncurator
are converted in the lanes that follow this commit.

* feat(memory): builtin engine CRUD, retained revisions and the curator's agent-user write

Slice 2 implementation (docs/prds/memory-engine-unification.md §1, §2, §6,
§7), on top of the contract seam in the previous commit.

- internal/infrastructure/memory: per-scope markdown heads with a JSON
  provenance marker, per-scope HISTORY.md retaining superseded and deleted
  states, Update superseding before replacing the live block, ErrStaleRevision
  via RecordUpdate.Expected, idempotent Forget, and the family scanner applied
  in Create and Update. Scope directories are hex(sha256(Scope.Key())).
- internal/infrastructure/sessioncurator: the participant is derived from the
  session's user-role senders; exactly one writes agent-user scope, zero or
  several write nothing and return ActionSkipped naming the count.
- docs/architecture: the settled decision recorded in migration-decisions §5
  (superseding the 2026-09-02 marker entry) and agent-system.md pointing at it.

Draft: to be squashed and reviewed before publication.

* test(curator): conform the registry's fake memory engine to the CRUD contract

The fake still implemented the pre-CRUD Store surface (Write taking an
Observation, List/Forget keyed by a bare string), so the package's test
binary no longer compiles: fakeMemoryEngine is missing Create, Get, Update
and Revisions, and domainmemory.Observation no longer exists. That took
`go fix ./...`, `go vet ./...` and `go test ./...` down with it, which is
what `task check` runs first.

Missed by the same commit that updated internal/infrastructure/sessioncurator's
own fake to the new interface. Repaired on the fake, not by loosening the
interface: the fake is the only thing out of date.

* fix(sessioncurator): refuse to attribute a session with unidentified user messages

sessionParticipant skipped user-role messages with an empty SenderID and
counted only the identified ones, so a session holding one identified sender
plus any number of unidentified user messages yielded that sender as the
single participant. Those messages are still part of the excerpt the
extraction model reads, so facts derived from content of unknown provenance
were written into the identified sender's agent-user scope: user A's memory
can end up crediting content that may belong to someone else. It is exactly
the case the function's own doc comment claims to handle, which it only
caught when no message at all was identified.

An unidentified user message now makes the session unattributable outright,
beside any number of identified senders, and the skip action's reason names
the count instead of reporting a sender count that no longer explains the
skip. Assistant-role messages are unaffected: they carry the bot, so they
still cannot manufacture a participant.

Fixed in sessionParticipant rather than at the write, because it is the
single producer of the attribution decision: it now returns the participant
or the reason there is none, so the caller cannot print a reason that
disagrees with what it decided. The fail-closed path (nothing written, one
ActionSkipped) and the extraction flow are unchanged.

* fix(memory): carry record content through the block format intact

builtin.Store's format ends a block at a blank line and starts a section at a
"## " line, but storeContent only trimmed the content and rejected an empty
one. Content holding either delimiter was accepted, written, and answered back
to the caller in full -- then read back truncated at the delimiter by Get,
List, or a process restart, with the remainder unaddressable and no error
anywhere along the path.

Fixed at the producer, in the render/parse pair that owns the block format:
renderBlock escapes the two delimiter shapes (an empty line becomes a single
backslash, a line already starting with a backslash or "## " gains one) and
parseBlocks undoes it on the way back, so a record's content survives the
write whole. Rejecting the content instead would have made an ordinary
multi-paragraph note or a quoted heading unrepresentable; escaping every line
would have changed the on-disk shape of every record. Only the lines that need
it are escaped, so a one-paragraph record is byte-identical to what the
unfixed engine wrote and the document stays hand-editable.

Reachable from both producers: the chat tool's note and the curator's
model-extracted fact are free text, and neither is constrained to one
paragraph.

Tests: TestBuiltinEngineRenderedBlockRoundTripsContent asserts
parseBlocks(renderBlock(c)) == c for eleven content shapes, and
TestBuiltinEngineContentSurvivesTheWriteAndAReopen carries the same shapes
through Create, Get, and a second engine reading the file a restart would.

* fix(memory): compare Update's expected revision against the file, not the cache

Update read the record to compare against out of findBlock(stores.live, ...)
-- the engine's cached copy of the scope document -- and checked Expected
against that copy. This engine caches one document per scope and rewrites it
whole, so a second engine over the same scope file (archied and the gateway,
or a restart) holds a copy taken when it last read the file: its view of the
record's revision is superseded, Expected matches the revision it remembers,
ErrStaleRevision never fires, and the rewrite that follows writes that stale
copy back over the file -- discarding the other writer's update and every
record written since the copy was loaded. The field existed and could not
fire.

Reload the scope document before anything is decided from it, so the check
answers about the revision actually on disk and the rewrite starts from the
current document. That is the smaller of the two fixes the PRD leaves open
("Not determined": a reload-before-mutate or an advisory lock); no locking
scheme, no retry loop, and no change to the read paths.

Still unsolved, and deliberately: the whole-document rewrite is
last-writer-wins between the reload and the write, so two engines writing one
scope concurrently can still lose one of the two writes, and HISTORY.md is
rewritten from the cached copy the same way.

Tests: TestBuiltinEngineSecondEngineCannotReuseAStaleRevision asserts the
second engine gets ErrStaleRevision and the first writer's state survives;
TestBuiltinEngineUpdateDoesNotEraseRecordsItHasNotLoaded asserts the record
written after the second engine loaded is still there after its update.

* fix(memory): emit the scanner's warning instead of dropping it

scanContent returned nil for every result that was not ThreatBlock, so a
ThreatWarn result -- documented in the scanner as "allow the write but emit a
warning", and what every sensitive-data match produces (API keys, passwords,
tokens, private keys) -- was stored with no audit signal anywhere. The
comment justified the silence by pointing at Bind, but Bind's argument is
about Registrar.Events: internal/events declares no memory kind and the
composition binds no memory event sink, so an invented kind would look wired
and not be. A scanner warning is a diagnostic rather than an event, and there
was nothing else to carry it: the registrar had no logger.

Bind now takes Registrar.Log, optional and mirroring the curator family's
Registrar for the same reason, scanContent emits the warn-level result
through it before the write is allowed, and bootstrap passes the daemon's
logger so the line reaches archied's configured sinks instead of only the
process default. The write is still allowed: only the silence changed.

Tests: TestBuiltinEngineScanWarnsWithoutRefusingTheWrite runs Create and
Update over warn, clean and block content, asserting the write lands either
way, that a warn-level scan result logs exactly one warning carrying the
scanner's own message, that clean content logs none, and that a refused
block-level match logs none.

* test(memory): carry every content shape through Update and Revisions

Update replaces the block it read out of the document by that text, and
retains the superseded state in HISTORY.md, so an escaped block has to be
found, superseded and retained exactly like an unescaped one. Covered here
rather than left to the Create path: without the escaping, every shape in
this table came back truncated after an update, including the plain
one-paragraph record whose replacement holds a blank line.

The test asserts values only: the updated record as a second engine reads it
off disk, and the superseded content as Revisions returns it.

* style(memory): apply the linter's gofumpt form to the Update revision error

golangci-lint's bundled gofumpt wants the closing paren of that wrapped
fmt.Errorf on its own line; the standalone gofumpt binary the fmt task uses
(0.7.0, a different version) leaves it as it was. The text is unchanged from
before the engine fixes in this branch, so this is pre-existing drift inside a
function these commits already touch -- taken so the file is clean under both
formatters rather than only one.
This commit is contained in:
sam authored and GitHub committed 2026-09-19 11:13:21 +10:00
1 parent cd9399f705
commit 7ee5025d3c
23 files changed
+3578 -607

No files matched your search

+5 -3
View File
@@ -95,9 +95,11 @@ because both users interact with the same Agent.
Shared memory does not erase provenance. Every memory record retains its scope,
author or acting identity, originating user when applicable, source, and
revision history. The exact rules for selecting a write scope, granting access
to global and user-wide memory, and authorizing explicit cross-scope sharing
require a focused memory design decision.
revision history. The rules for selecting a write scope, granting access
to global and user-wide memory, and authorizing explicit cross-scope sharing are
settled by [`docs/prds/memory-engine-unification.md`](../prds/memory-engine-unification.md)
and recorded in [`migration-decisions.md`](migration-decisions.md#5-memory-placement-and-storage)
section 5.
An Agent interaction may answer conversationally, use a tool or capability, read
or send messages, retain information, or propose durable workflow-backed work.
+64 -7
View File
@@ -242,14 +242,18 @@ The four required scopes are fixed:
- Agent-user relationship.
A focused review of `internal/memory`, its providers, consumers, persistence,
and tests must decide:
and tests had these decisions:
- `internal/domain/memory` versus ownership within the Agent domain;
- the authoritative memory record and revision model;
- provider and infrastructure boundaries;
- retrieval and access enforcement;
- migration of existing memory data;
- provenance representation.
- `internal/domain/memory` versus ownership within the Agent domain — settled
(2026-09-02, below);
- the authoritative memory record and revision model — settled by the PRD
(2026-09-16, below);
- provider and infrastructure boundaries — settled by the PRD (2026-09-16,
below);
- retrieval and access enforcement — settled by the PRD (2026-09-16, below);
ranked retrieval remains open (below);
- migration of existing memory data — open: existing data is not migrated;
- provenance representation — settled by the PRD (2026-09-16, below).
Conversation branches do not create another memory scope. A memory action has
the same scoped effect regardless of which Conversation originated it.
@@ -295,6 +299,59 @@ memory model in general or a future non-file engine's model:
invalid `Kind` fails `Write` with the same validation error `memory_edit`
already returns for an invalid section today, not a new error shape.
**Memory engine: scopes, CRUD with retained revisions, and store placement —
DECIDED (2026-09-16).** Supersedes the 2026-09-02 "Authoritative record and
revision model for the builtin engine" entry above, whose `<!--mem:<ulid>-->`
live marker cannot carry provenance. Authority:
[`docs/prds/memory-engine-unification.md`](../prds/memory-engine-unification.md)
(Approved); the contract is `internal/domain/memory/contract.go`. This register
supersedes; it does not erase.
- **Four typed scopes, caller-named.** `internal/domain/memory` is addressed by
`Scope{Kind, Agent, User}` over `global`, `agent`, `user` and `agent-user`,
with `Validate` rejecting a component present where it is not required. A
caller names the read and write sets (`Subject.Scopes()` /
`WritableScopes()`; global is never model-writable) and the engine applies no
policy of its own: **access control is naming a scope, and nothing else.** An
engine that enforced policy would need to know about channels, sessions and
bindings, and every backend would reimplement it. `Scope.Key()` length-prefixes
each component, so two scopes can never share a key even when an opaque
channel-native id embeds the separator.
- **The caller resolves identity; the engine never does.** Resolution happens in
the gateway turn path, where the native sender id and the session are both in
hand, and is supplied per channel by the composition. A webhook's sender id is
a route path, not a person, and treating it as one would give a URL an
identity. An absent user id yields global and agent scopes only: "show none of
it", never a wider fallback.
- **CRUD supersedes, never overwrites.** `Store` is `Create`/`Get`/`Query`/
`List`/`Update`/`Forget`/`Revisions` over the four scopes. `Update` appends the
superseded state to the scope's `HISTORY.md` before replacing the live block;
`Update.Expected` names the revision being replaced and returns
`ErrStaleRevision` on mismatch, making a lost update between two processes
sharing one scope file detectable. `Forget` records a deleted revision before
removing the live block, so a record's provenance outlives its content; `Get`
and `Forget` take a scope, because an id alone does not locate one.
- **Provenance representation.** The live marker becomes JSON, since a
`<!--mem:<ulid>-->` line cannot carry provenance, and `Record` retains the
provenance `agent-system.md` requires — scope, author, originating user,
source, and revision history. `Metadata` is dropped: it never round-tripped.
- **Store and scanner placement.** The markdown store relocates to
`internal/infrastructure/memory/builtin`, its correct home under
`organisation.md`'s "a package owns its on-disk format end-to-end". The content
scanner moves into `internal/domain/memory` and is applied in `Create` and
`Update` — the single choke point every producer crosses, so curator-extracted
content is covered too.
**Open in this section.** Ranked retrieval is deferred: `Query.Text` stays in
the contract, unused by the prompt path. Existing memory data is not migrated:
`<workDir>/memory` and `<workDir>/memory-engine` are left on disk, unreferenced.
The PRD leaves these undetermined and they remain open here: group-chat
extraction, where the curator fails closed on mixed-sender sessions; two
processes (`archied` and `archie-gateway`) holding one scope file, where
`Update.Expected` detects but does not prevent a lost block; unbounded
`HISTORY.md` with no compaction policy; whether the per-scope size bound is
right; and whether the dashboard should carry a user identity.
### 6. Runtime and process boundaries — DECIDED
`archied` is the native resident orchestration and interactive-chat process.
+3 -1
View File
@@ -1112,7 +1112,9 @@ func (b *boot) setupMemory() error {
// same files while both exist.
func (b *boot) setupMemoryEngine() error {
cfg, log := b.cfg, b.log
registry := domainmemory.NewRegistry(domainmemory.Registrar{})
// The logger is what carries an engine's scanner warning: warn allows
// the write, so the log line is the whole audit trail for it.
registry := domainmemory.NewRegistry(domainmemory.Registrar{Log: log})
switch cfg.Memory.Engine {
case infraMemory.EngineName, "":
+1 -1
View File
@@ -39,8 +39,8 @@ import (
"github.com/samcharles93/archie-core/internal/gateway"
"github.com/samcharles93/archie-core/internal/infrastructure/configuration"
"github.com/samcharles93/archie-core/internal/infrastructure/configuration/overlay"
"github.com/samcharles93/archie-core/internal/infrastructure/memory/builtin"
"github.com/samcharles93/archie-core/internal/logging"
"github.com/samcharles93/archie-core/internal/memory/builtin"
"github.com/samcharles93/archie-core/internal/plugin"
"github.com/samcharles93/archie-core/internal/secret"
"github.com/samcharles93/archie-core/internal/storage"
+19 -3
View File
@@ -61,18 +61,34 @@ func (fakeMemoryEngine) Health(context.Context) domainmemory.Health {
return domainmemory.Health{Status: domainmemory.HealthHealthy}
}
func (fakeMemoryEngine) Stop(context.Context) error { return nil }
func (fakeMemoryEngine) Write(context.Context, domainmemory.Observation) (domainmemory.Record, error) {
func (fakeMemoryEngine) Create(context.Context, domainmemory.NewRecord) (domainmemory.Record, error) {
return domainmemory.Record{}, nil
}
func (fakeMemoryEngine) Get(context.Context, domainmemory.Scope, domainmemory.RecordID) (domainmemory.Record, error) {
return domainmemory.Record{}, domainmemory.ErrNotFound
}
func (fakeMemoryEngine) Query(context.Context, domainmemory.Query) ([]domainmemory.Record, error) {
return nil, nil
}
func (fakeMemoryEngine) List(context.Context, string) ([]domainmemory.Record, error) {
func (fakeMemoryEngine) List(context.Context, domainmemory.Scope) ([]domainmemory.Record, error) {
return nil, nil
}
func (fakeMemoryEngine) Update(context.Context, domainmemory.RecordUpdate) (domainmemory.Record, error) {
return domainmemory.Record{}, domainmemory.ErrNotFound
}
func (fakeMemoryEngine) Forget(context.Context, domainmemory.Scope, domainmemory.RecordID) error {
return nil
}
func (fakeMemoryEngine) Revisions(context.Context, domainmemory.Scope, domainmemory.RecordID) ([]domainmemory.Revision, error) {
return nil, nil
}
func (fakeMemoryEngine) Forget(context.Context, string) error { return nil }
// testConversations is a fake ConversationSource with one session and no
// messages -- enough for the presence/absence tests, which never call
+327 -66
View File
@@ -1,24 +1,32 @@
// Package memory is the memory engine family: a pluggable backend for
// durable observations about a session or identity, so an external memory
// service (Honcho, or a future alternative) can be bolted on without
// touching the agent loop. See epic archie-core-1786637499161-356-e424e40d.
// Package memory is the memory engine family: the authoritative store for
// durable observations, addressed by typed scope. See
// docs/prds/memory-engine-unification.md for the decision this contract
// implements.
//
// One engine, four scopes. A caller names the scopes it may read or write
// (Scope, Subject) and the engine applies no policy of its own: an engine
// that enforced access would need to know about channels, sessions and
// bindings, and every backend would reimplement it. Isolation then falls out
// mechanically, because ScopeAgent{a} and ScopeAgent{b} are different
// storage keys.
//
// The family follows the strict plugin engine rule: a typed contract
// (MemoryEngine) with real domain operations (write, query, list, forget —
// not just Name/Version), an owning registry (Registry) with
// (MemoryEngine) with real domain operations (create, read, update, delete,
// revisions — not just Name/Version), an owning registry (Registry) with
// start/health/stop and shutdown ordering, and narrow typed host access
// (Registrar) — an engine never receives the daemon or an untyped hook map.
//
// This package defines the contract only. internal/memory's existing
// MemoryProvider and Manager predate the domain migration and are the
// current runtime path; they are not extended in place (see
// archie-core-1786637490069-11-e245c170's notes) and are superseded by this
// family as engines land against it.
// This package owns the contract and the family's policy that is independent
// of any backend (the content scanner). internal/memory's MemoryProvider and
// Manager predate the domain migration and are the legacy runtime path,
// deleted in slice 5 of the same PRD; nothing here depends on them.
package memory
import (
"context"
"errors"
"fmt"
"strconv"
"time"
)
@@ -59,23 +67,186 @@ type Lifecycle interface {
Stop(ctx context.Context) error
}
// Store is the real domain operations every memory backend implements —
// what the epic requires beyond identity metadata. Split out from
// MemoryEngine so the two concerns (storage vs. identity/lifecycle) stay
// separately named and independently testable.
// ScopeKind names one of the four durable memory scopes
// (docs/architecture/agent-system.md, "Memory scopes").
type ScopeKind string
const (
// ScopeGlobal is shared across the application. Never model-writable:
// writing it is an operator action. It is part of the contract because
// the read set includes it.
ScopeGlobal ScopeKind = "global"
// ScopeAgent is private to one Agent. Agent-wide is not a synonym for
// shared; ScopeAgent{a} and ScopeAgent{b} are different keys.
ScopeAgent ScopeKind = "agent"
// ScopeUser is one user, across every Agent serving them.
ScopeUser ScopeKind = "user"
// ScopeAgentUser is one Agent's relationship with one user: the only
// home for relationship facts, and the scope the session curator
// writes.
ScopeAgentUser ScopeKind = "agent-user"
)
// AgentID and IdentityID are opaque. This package never resolves them,
// canonicalises them, or invents one from a channel: resolution happens
// where the channel's native identity is in hand (the gateway turn path),
// because a webhook's sender id is a route path and treating it as a person
// would give a URL an identity.
type (
AgentID string
IdentityID string
)
// Scope addresses one region of memory. Which components are required is a
// function of Kind, and Validate is the only place that rule lives.
type Scope struct {
Kind ScopeKind
Agent AgentID
User IdentityID
}
// Validate rejects an unknown kind, a missing component, and — just as
// importantly — a component present where it is not required, so no two
// spellings address the same region.
func (s Scope) Validate() error {
switch s.Kind {
case ScopeGlobal:
if s.Agent != "" || s.User != "" {
return fmt.Errorf("memory scope global: takes no agent or user, got agent %q user %q", s.Agent, s.User)
}
case ScopeAgent:
if s.Agent == "" {
return errors.New("memory scope agent: agent must not be empty")
}
if s.User != "" {
return fmt.Errorf("memory scope agent: takes no user, got %q", s.User)
}
case ScopeUser:
if s.User == "" {
return errors.New("memory scope user: user must not be empty")
}
if s.Agent != "" {
return fmt.Errorf("memory scope user: takes no agent, got %q", s.Agent)
}
case ScopeAgentUser:
if s.Agent == "" || s.User == "" {
return fmt.Errorf("memory scope agent-user: agent and user both required, got agent %q user %q", s.Agent, s.User)
}
default:
return fmt.Errorf("memory scope: unknown kind %q", s.Kind)
}
return nil
}
// Key returns the scope's canonical storage key. Each component is
// length-prefixed, so two different scopes can never share a key even when
// an id embeds the separator byte — which matters because AgentID and
// IdentityID are opaque and come from channels this package does not
// control.
//
// An invalid scope returns "", which a storage key can never legitimately
// be: callers must Validate first. Every engine path does.
func (s Scope) Key() string {
switch s.Kind {
case ScopeGlobal:
return "global"
case ScopeAgent:
return "agent:" + lengthPrefixed(string(s.Agent))
case ScopeUser:
return "user:" + lengthPrefixed(string(s.User))
case ScopeAgentUser:
return "agent-user:" + lengthPrefixed(string(s.Agent)) + ":" + lengthPrefixed(string(s.User))
default:
return ""
}
}
// String renders the scope for logs and error messages in Key's canonical
// form.
func (s Scope) String() string { return s.Key() }
// lengthPrefixed renders s as "<byteLen>:<s>", so a component containing ':'
// (or any other byte) cannot be confused with the boundary between
// components.
func lengthPrefixed(s string) string {
return strconv.Itoa(len(s)) + ":" + s
}
// Subject is a turn's resolved identity: one value, supplied by the
// composition per channel, never inferred by the engine.
type Subject struct {
// AgentID is the Agent serving the turn. Required in practice; an empty
// AgentID simply yields no agent-addressed scopes.
AgentID AgentID
// UserID is the initiating user's identity. Empty when the channel
// carries no per-person identity (the dashboard, a webhook) — which
// means "show none of it", never "fall back to a wider scope".
UserID IdentityID
}
// Scopes returns the scopes a read for this subject names, narrowest first:
// agent-user, agent, user, then global last, because global is the only
// scope every subject shares.
//
// The read path hands exactly this set to the engine and the engine unions
// nothing beyond it.
func (s Subject) Scopes() []Scope {
return append(s.WritableScopes(), Scope{Kind: ScopeGlobal})
}
// WritableScopes returns the scopes a write for this subject may name —
// Scopes() minus global, which is never model-writable. A component the
// subject does not carry removes every scope that requires it.
func (s Subject) WritableScopes() []Scope {
var scopes []Scope
if s.AgentID != "" && s.UserID != "" {
scopes = append(scopes, Scope{Kind: ScopeAgentUser, Agent: s.AgentID, User: s.UserID})
}
if s.AgentID != "" {
scopes = append(scopes, Scope{Kind: ScopeAgent, Agent: s.AgentID})
}
if s.UserID != "" {
scopes = append(scopes, Scope{Kind: ScopeUser, User: s.UserID})
}
return scopes
}
// Store is the real domain operations every memory backend implements.
// Split out from MemoryEngine so the two concerns (storage vs.
// identity/lifecycle) stay separately named and independently testable.
//
// Every method that touches one record takes the record's Scope: a record id
// alone does not locate a scope, and an engine that had to search for one
// would be reimplementing the caller's addressing.
type Store interface {
// Write records one observation and returns the stored Record,
// including the identifier Forget later needs.
Write(ctx context.Context, obs Observation) (Record, error)
// Query returns records relevant to q, most relevant or most recent
// first — ordering is engine-defined and not part of this contract.
// Create records one new memory and returns the stored Record,
// including the identifier Get/Update/Forget/Revisions later need.
Create(ctx context.Context, in NewRecord) (Record, error)
// Get returns one record by id within scope. A missing record is
// ErrNotFound.
Get(ctx context.Context, scope Scope, id RecordID) (Record, error)
// Query returns the live records of q.Scopes, grouped in the order
// those scopes were named and ordered within a scope by the engine
// (most recent first is the expected choice). Heads only: superseded
// states are Revisions.
Query(ctx context.Context, q Query) ([]Record, error)
// List returns every record for identity, for inspection and
// deletion flows rather than relevance-ranked recall.
List(ctx context.Context, identity string) ([]Record, error)
// Forget permanently removes one record. Forgetting an id that does
// not exist is not an error: the end state (id absent) already holds.
Forget(ctx context.Context, id string) error
// List returns every live record of one scope, for inspection and
// deletion flows rather than relevance-ranked recall. Heads only.
List(ctx context.Context, scope Scope) ([]Record, error)
// Update supersedes a record's content and returns the new state. The
// superseded state is retained and readable through Revisions.
// Unaddressed, or already superseded when RecordUpdate.Expected is
// set, returns ErrNotFound or ErrStaleRevision respectively.
Update(ctx context.Context, in RecordUpdate) (Record, error)
// Forget removes one record's live state, retaining a deleted revision
// so its provenance outlives its content. Forgetting a record that
// does not exist is not an error: the end state (id absent) already
// holds.
Forget(ctx context.Context, scope Scope, id RecordID) error
// Revisions returns one record's retained states, oldest first,
// including the deleted revision Forget records. A record that never
// existed is not an error: the result is empty.
Revisions(ctx context.Context, scope Scope, id RecordID) ([]Revision, error)
}
// MemoryEngine is the typed contract every memory backend implements.
@@ -93,64 +264,154 @@ type MemoryEngine interface {
Bind(host Registrar)
}
// Observation is one fact to remember, submitted for writing.
type Observation struct {
// Identity is whose memory this is about (a session id, a user id —
// engine-defined, but always required so Query/List/Forget have a
// scope to operate within).
Identity string
// Kind is a freeform category ("preference", "fact", "correction").
Kind string
// Content is the observation itself.
// RecordID identifies one record within its scope, stable for the record's
// whole life: assigned once at Create, independent of content, so an edit
// somewhere else can never orphan an id.
type RecordID string
// NewRecord is one memory submitted for creation. Revision, timestamps and
// id are stamped by the engine; Scope is what the record will be addressed
// by for the rest of its life.
type NewRecord struct {
Scope Scope
Kind string
Content string
// At is when the observation was made. Zero means the engine should
// stamp it with the current time.
At time.Time
// Metadata carries engine-agnostic key/value context alongside the
// observation (e.g. source, confidence).
Metadata map[string]string
// Author is the acting identity that wrote the record — an agent id, or
// a producer's own name ("operator", "session-memory curator"). Free
// text: the engine retains it, it does not interpret it.
Author string
// OriginUser is the user whose interaction produced the record, when
// one is known. Distinct from Scope.User: a curator writing
// agent-user scope has both, but a record about a user may be written
// into agent scope by an operator.
OriginUser IdentityID
// Source is the producer's own label for where the record came from
// (a tool name, a curator pass, a session id). Free text.
Source string
}
// Validate checks the fields every engine needs regardless of backend:
// Identity scopes every other operation, and Content is the whole point
// of writing.
func (o Observation) Validate() error {
if o.Identity == "" {
return errors.New("memory observation: identity must not be empty")
// Validate checks what every engine needs before writing: an addressable
// scope and content. Kind's format belongs to the backend (the builtin
// engine maps it to a section name), so it is validated there.
//
// Implementations must also scan Content before persisting it — see
// ScanContent and docs/prds/memory-engine-unification.md §7.
func (n NewRecord) Validate() error {
if err := n.Scope.Validate(); err != nil {
return err
}
if o.Content == "" {
return errors.New("memory observation: content must not be empty")
if n.Content == "" {
return errors.New("memory record: content must not be empty")
}
return nil
}
// Query asks an engine to recall observations relevant to Text within
// Identity's scope.
// Query asks an engine to recall records within the caller-named scopes.
type Query struct {
Identity string
Text string
// Scopes is the read set, narrowest first (Subject.Scopes()). The
// engine unions exactly these and applies no policy of its own.
Scopes []Scope
// Text filters by content substring when non-empty. It stays unused by
// the prompt path; ranked retrieval is a deliberately open question.
Text string
// Limit bounds the number of records returned. Zero lets the engine
// choose a default; there is no way to request "unlimited" — List
// exists for that.
Limit int
}
// Validate checks the field every engine needs to scope a query: Identity.
// Text may be empty (some engines support a scope-only recall), so it is
// not required here.
// Validate rejects an empty or invalid read set, and a negative limit.
func (q Query) Validate() error {
if q.Identity == "" {
return errors.New("memory query: identity must not be empty")
if len(q.Scopes) == 0 {
return errors.New("memory query: at least one scope is required")
}
for _, scope := range q.Scopes {
if err := scope.Validate(); err != nil {
return fmt.Errorf("memory query: %w", err)
}
}
if q.Limit < 0 {
return fmt.Errorf("memory query: limit must not be negative, got %d", q.Limit)
}
return nil
}
// Record is one stored observation as returned by Write, Query, or List.
type Record struct {
ID string
Identity string
Kind string
Content string
At time.Time
Metadata map[string]string
// RecordUpdate supersedes one record's content. It deliberately cannot name
// a Scope: a record may never move between scopes, so the API makes that
// unrepresentable rather than validating it.
type RecordUpdate struct {
// Scope and ID address the record being superseded.
Scope Scope
ID RecordID
// Content replaces the live content. Required.
Content string
// Expected optionally names the revision this update supersedes. Zero
// means "whatever is live". A mismatch returns ErrStaleRevision, which
// matters because two processes (archied and archie-gateway) can hold
// the same scope's file.
Expected int
// Author and Source are refreshed on the new revision, with the same
// meaning as in NewRecord.
Author string
Source string
}
// Validate checks the address and the replacement content.
func (u RecordUpdate) Validate() error {
if err := u.Scope.Validate(); err != nil {
return err
}
if u.ID == "" {
return errors.New("memory record update: id must not be empty")
}
if u.Content == "" {
return errors.New("memory record update: content must not be empty")
}
if u.Expected < 0 {
return fmt.Errorf("memory record update: expected revision must not be negative, got %d", u.Expected)
}
return nil
}
// Record is one stored memory as returned by Create, Get, Query, List or
// Update. It carries the provenance agent-system.md requires be retained:
// scope, kind, content, revision, author, originating user, source, and
// timestamps.
type Record struct {
ID RecordID
Scope Scope
Kind string
Content string
Revision int
Author string
OriginUser IdentityID
Source string
CreatedAt time.Time
UpdatedAt time.Time
}
// Revision is one retained state of a record: a state that was live and has
// been superseded, or the deletion marker Forget records. The record's own
// Revision field is the revision this state carried while live.
type Revision struct {
Record Record
// SupersededAt is when this state stopped being live (replaced by an
// Update, or removed by a Forget).
SupersededAt time.Time
// Deleted marks the revision Forget recorded: Record.Content is the
// last live content, retained so provenance outlives it.
Deleted bool
}
// Store errors. Both cross a producer/consumer boundary, and both are
// expected conditions rather than failures: a missing record and a losing
// race between two processes holding one scope.
var (
// ErrNotFound is returned when a record does not exist in the named
// scope. Get and Update return it; Forget and Revisions do not, because
// their end state already holds.
ErrNotFound = errors.New("memory record not found")
// ErrStaleRevision is returned when RecordUpdate.Expected names a
// revision that is no longer live.
ErrStaleRevision = errors.New("memory record revision is stale")
)
+260 -16
View File
@@ -1,39 +1,271 @@
package memory
import "testing"
import (
"strings"
"testing"
)
func TestObservationValidateRequiresIdentityAndContent(t *testing.T) {
func TestScopeValidate(t *testing.T) {
t.Parallel()
tests := []struct {
name string
obs Observation
wantErr bool
scope Scope
wantErr string // "" means valid; otherwise a substring of the error
}{
{"empty identity", Observation{Content: "x"}, true},
{"empty content", Observation{Identity: "u1"}, true},
{"both empty", Observation{}, true},
{"valid", Observation{Identity: "u1", Content: "x"}, false},
{"global", Scope{Kind: ScopeGlobal}, ""},
{"agent", Scope{Kind: ScopeAgent, Agent: "a1"}, ""},
{"user", Scope{Kind: ScopeUser, User: "u1"}, ""},
{"agent-user", Scope{Kind: ScopeAgentUser, Agent: "a1", User: "u1"}, ""},
{"unknown kind", Scope{Kind: "session"}, "unknown kind"},
{"empty kind", Scope{}, "unknown kind"},
{"global rejects agent", Scope{Kind: ScopeGlobal, Agent: "a1"}, "takes no agent or user"},
{"global rejects user", Scope{Kind: ScopeGlobal, User: "u1"}, "takes no agent or user"},
{"agent needs an agent", Scope{Kind: ScopeAgent}, "agent must not be empty"},
{"agent rejects a user", Scope{Kind: ScopeAgent, Agent: "a1", User: "u1"}, "takes no user"},
{"user needs a user", Scope{Kind: ScopeUser}, "user must not be empty"},
{"user rejects an agent", Scope{Kind: ScopeUser, Agent: "a1", User: "u1"}, "takes no agent"},
{"agent-user needs an agent", Scope{Kind: ScopeAgentUser, User: "u1"}, "both required"},
{"agent-user needs a user", Scope{Kind: ScopeAgentUser, Agent: "a1"}, "both required"},
{"agent-user needs both", Scope{Kind: ScopeAgentUser}, "both required"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
err := tt.obs.Validate()
if (err != nil) != tt.wantErr {
t.Fatalf("Validate() = %v, wantErr %v", err, tt.wantErr)
err := tt.scope.Validate()
switch {
case tt.wantErr == "" && err != nil:
t.Fatalf("Validate() = %v, want nil", err)
case tt.wantErr != "" && err == nil:
t.Fatalf("Validate() = nil, want an error mentioning %q", tt.wantErr)
case tt.wantErr != "" && !strings.Contains(err.Error(), tt.wantErr):
t.Fatalf("Validate() = %v, want an error mentioning %q", err, tt.wantErr)
}
})
}
}
func TestQueryValidateRequiresIdentity(t *testing.T) {
func TestScopeKeyIsCanonical(t *testing.T) {
t.Parallel()
if err := (Query{}).Validate(); err == nil {
t.Fatal("Validate() = nil for empty identity, want error")
tests := []struct {
name string
scope Scope
want string
}{
{"global", Scope{Kind: ScopeGlobal}, "global"},
{"agent", Scope{Kind: ScopeAgent, Agent: "telegram"}, "agent:8:telegram"},
{"user", Scope{Kind: ScopeUser, User: "u1"}, "user:2:u1"},
{"agent-user", Scope{Kind: ScopeAgentUser, Agent: "a1", User: "u1"}, "agent-user:2:a1:2:u1"},
// An invalid scope has no storage key; "" is the signal, never a key
// a valid scope could produce.
{"invalid kind", Scope{Kind: "session", Agent: "a1"}, ""},
{"missing component", Scope{Kind: ScopeAgentUser, Agent: "a1"}, "agent-user:2:a1:0:"},
}
if err := (Query{Identity: "u1"}).Validate(); err != nil {
t.Fatalf("Validate() = %v for identity-only query, want nil (Text may be empty)", err)
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
if got := tt.scope.Key(); got != tt.want {
t.Fatalf("Key() = %q, want %q", got, tt.want)
}
})
}
}
// TestScopeKeyCollisionFree is the property Key's length prefix exists for:
// AgentID and IdentityID are channel-native and opaque, so a separator-based
// key would let two different scopes share one storage key. Every scope over
// ids that embed the separator must still key distinctly.
func TestScopeKeyCollisionFree(t *testing.T) {
t.Parallel()
ids := []string{"", "a", "a:", ":a", "a:1", "1", "1:a", "agent-user:1:a"}
distinct := map[Scope]bool{{Kind: ScopeGlobal}: true}
for _, agent := range ids {
for _, user := range ids {
distinct[Scope{Kind: ScopeAgentUser, Agent: AgentID(agent), User: IdentityID(user)}] = true
distinct[Scope{Kind: ScopeAgent, Agent: AgentID(agent)}] = true
distinct[Scope{Kind: ScopeUser, User: IdentityID(user)}] = true
}
}
seen := make(map[string]Scope, len(distinct))
for scope := range distinct {
if err := scope.Validate(); err != nil {
continue // an incomplete scope has no key to collide with
}
key := scope.Key()
if key == "" {
t.Fatalf("valid scope %+v has an empty key", scope)
}
if other, dup := seen[key]; dup {
t.Fatalf("scopes %+v and %+v share key %q", other, scope, key)
}
seen[key] = scope
}
}
func TestSubjectScopesNarrowestFirst(t *testing.T) {
t.Parallel()
tests := []struct {
name string
subject Subject
want []string
}{
{
name: "agent and user",
subject: Subject{AgentID: "a1", UserID: "u1"},
want: []string{"agent-user:2:a1:2:u1", "agent:2:a1", "user:2:u1", "global"},
},
{
// A channel with no per-person identity (dashboard, webhook)
// gets agent and global only -- never a wider scope to stand in
// for the one it does not know.
name: "agent only",
subject: Subject{AgentID: "a1"},
want: []string{"agent:2:a1", "global"},
},
{
name: "user only",
subject: Subject{UserID: "u1"},
want: []string{"user:2:u1", "global"},
},
{
name: "neither",
subject: Subject{},
want: []string{"global"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got := keys(tt.subject.Scopes())
if len(got) != len(tt.want) {
t.Fatalf("Scopes() = %v, want %v", got, tt.want)
}
for i := range got {
if got[i] != tt.want[i] {
t.Fatalf("Scopes() = %v, want %v", got, tt.want)
}
}
})
}
}
func TestSubjectWritableScopesExcludeGlobal(t *testing.T) {
t.Parallel()
subject := Subject{AgentID: "a1", UserID: "u1"}
writable := keys(subject.WritableScopes())
if len(writable) != 3 {
t.Fatalf("WritableScopes() = %v, want three scopes", writable)
}
for _, key := range writable {
if key == "global" {
t.Fatal("WritableScopes() includes global, which is never model-writable")
}
}
// Every writable scope is readable: the read set is a superset.
readable := map[string]bool{}
for _, key := range keys(subject.Scopes()) {
readable[key] = true
}
for _, key := range writable {
if !readable[key] {
t.Fatalf("writable scope %q is not in the read set", key)
}
}
}
func keys(scopes []Scope) []string {
out := make([]string, 0, len(scopes))
for _, scope := range scopes {
out = append(out, scope.Key())
}
return out
}
func TestNewRecordValidate(t *testing.T) {
t.Parallel()
valid := Scope{Kind: ScopeAgent, Agent: "a1"}
tests := []struct {
name string
in NewRecord
wantErr string
}{
{"valid", NewRecord{Scope: valid, Content: "x"}, ""},
{"valid without kind", NewRecord{Scope: valid, Content: "x"}, ""},
{"empty content", NewRecord{Scope: valid}, "content must not be empty"},
{"invalid scope", NewRecord{Scope: Scope{Kind: ScopeAgent}, Content: "x"}, "agent must not be empty"},
{"no scope", NewRecord{Content: "x"}, "unknown kind"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
assertValidation(t, tt.in.Validate(), tt.wantErr)
})
}
}
func TestRecordUpdateValidate(t *testing.T) {
t.Parallel()
valid := Scope{Kind: ScopeAgent, Agent: "a1"}
tests := []struct {
name string
in RecordUpdate
wantErr string
}{
{"valid", RecordUpdate{Scope: valid, ID: "r1", Content: "x"}, ""},
{"valid with expected", RecordUpdate{Scope: valid, ID: "r1", Content: "x", Expected: 2}, ""},
{"empty id", RecordUpdate{Scope: valid, Content: "x"}, "id must not be empty"},
{"empty content", RecordUpdate{Scope: valid, ID: "r1"}, "content must not be empty"},
{"invalid scope", RecordUpdate{Scope: Scope{Kind: ScopeGlobal, Agent: "a1"}, ID: "r1", Content: "x"}, "takes no agent or user"},
{"negative expected", RecordUpdate{Scope: valid, ID: "r1", Content: "x", Expected: -1}, "must not be negative"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
assertValidation(t, tt.in.Validate(), tt.wantErr)
})
}
}
func TestQueryValidate(t *testing.T) {
t.Parallel()
agent := Scope{Kind: ScopeAgent, Agent: "a1"}
tests := []struct {
name string
q Query
wantErr string
}{
{"one scope", Query{Scopes: []Scope{agent}}, ""},
{"two scopes", Query{Scopes: []Scope{agent, {Kind: ScopeGlobal}}}, ""},
{"scope only, no text", Query{Scopes: []Scope{agent}}, ""},
{"with limit", Query{Scopes: []Scope{agent}, Limit: 5}, ""},
{"no scopes", Query{}, "at least one scope"},
{"invalid scope", Query{Scopes: []Scope{{Kind: ScopeAgentUser, Agent: "a1"}}}, "both required"},
{"negative limit", Query{Scopes: []Scope{agent}, Limit: -1}, "must not be negative"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
assertValidation(t, tt.q.Validate(), tt.wantErr)
})
}
}
@@ -44,3 +276,15 @@ func TestManifestValidateAcceptsZeroValue(t *testing.T) {
t.Fatalf("Validate() = %v, want nil: no field is required yet", err)
}
}
func assertValidation(t *testing.T, err error, wantErr string) {
t.Helper()
switch {
case wantErr == "" && err != nil:
t.Fatalf("Validate() = %v, want nil", err)
case wantErr != "" && err == nil:
t.Fatalf("Validate() = nil, want an error mentioning %q", wantErr)
case wantErr != "" && !strings.Contains(err.Error(), wantErr):
t.Fatalf("Validate() = %v, want an error mentioning %q", err, wantErr)
}
}
+11 -1
View File
@@ -1,6 +1,9 @@
package memory
import "time"
import (
"log/slog"
"time"
)
// Registrar is the narrow typed host access an engine receives at
// registration. Fields are typed contracts declared here — this package
@@ -16,6 +19,13 @@ type Registrar struct {
// Clock is injectable time. Always bound; nil at construction is
// replaced with the system clock.
Clock Clock
// Log is the engine-scoped diagnostic logger, optional so domain tests
// and embedders can construct a registrar without a logging sink (the
// curator family's Registrar carries the same field for the same
// reason). An engine emits here what it cannot emit as an event: the
// content scanner's warn level, which allows the write and is expected
// to leave a record of what it saw.
Log *slog.Logger
}
// EventSink publishes engine activity. Mirrors curator.EventSink: emission
+82 -12
View File
@@ -4,6 +4,8 @@ import (
"context"
"errors"
"slices"
"strconv"
"strings"
"sync"
"testing"
"time"
@@ -21,12 +23,18 @@ type fakeEngine struct {
bindPanic bool
view Registrar
records map[string]Record
nextID int
records map[RecordID]Record
revisions map[RecordID][]Revision
nextID int
}
func newFake(name string, manifest Manifest) *fakeEngine {
return &fakeEngine{name: name, manifest: manifest, records: map[string]Record{}}
return &fakeEngine{
name: name,
manifest: manifest,
records: map[RecordID]Record{},
revisions: map[RecordID][]Revision{},
}
}
func (f *fakeEngine) record(e string) {
@@ -46,33 +54,95 @@ func (f *fakeEngine) Bind(host Registrar) {
}
}
func (f *fakeEngine) Write(_ context.Context, obs Observation) (Record, error) {
// The fake implements the full Store so it satisfies MemoryEngine, but these
// tests exercise registration and lifecycle, not storage semantics -- the
// builtin engine's own suite is the store's real coverage. The in-memory
// behaviour here is deliberately minimal and not a reference implementation.
func (f *fakeEngine) Create(_ context.Context, in NewRecord) (Record, error) {
f.nextID++
id := obs.Identity + "-" + time.Now().Format("150405.000000000") + "-" + string(rune('a'+f.nextID))
rec := Record{ID: id, Identity: obs.Identity, Kind: obs.Kind, Content: obs.Content, At: obs.At, Metadata: obs.Metadata}
f.records[id] = rec
rec := Record{
ID: RecordID(f.name + "-" + strconv.Itoa(f.nextID)),
Scope: in.Scope,
Kind: in.Kind,
Content: in.Content,
Revision: 1,
Author: in.Author,
OriginUser: in.OriginUser,
Source: in.Source,
}
f.records[rec.ID] = rec
return rec, nil
}
func (f *fakeEngine) Get(_ context.Context, scope Scope, id RecordID) (Record, error) {
rec, ok := f.records[id]
if !ok || rec.Scope != scope {
return Record{}, ErrNotFound
}
return rec, nil
}
func (f *fakeEngine) Query(_ context.Context, q Query) ([]Record, error) {
wanted := make(map[Scope]bool, len(q.Scopes))
for _, scope := range q.Scopes {
wanted[scope] = true
}
var out []Record
for _, r := range f.records {
if r.Identity == q.Identity {
if q.Text != "" && !strings.Contains(r.Content, q.Text) {
continue
}
if wanted[r.Scope] {
out = append(out, r)
}
}
return out, nil
slices.SortFunc(out, func(a, b Record) int { return strings.Compare(string(a.ID), string(b.ID)) })
return limit(out, q.Limit), nil
}
func (f *fakeEngine) List(ctx context.Context, identity string) ([]Record, error) {
return f.Query(ctx, Query{Identity: identity})
func (f *fakeEngine) List(ctx context.Context, scope Scope) ([]Record, error) {
return f.Query(ctx, Query{Scopes: []Scope{scope}})
}
func (f *fakeEngine) Forget(_ context.Context, id string) error {
func (f *fakeEngine) Update(_ context.Context, in RecordUpdate) (Record, error) {
rec, ok := f.records[in.ID]
if !ok || rec.Scope != in.Scope {
return Record{}, ErrNotFound
}
if in.Expected != 0 && in.Expected != rec.Revision {
return Record{}, ErrStaleRevision
}
f.revisions[rec.ID] = append(f.revisions[rec.ID], Revision{Record: rec, Deleted: false})
rec.Content = in.Content
rec.Revision++
rec.Author = in.Author
rec.Source = in.Source
f.records[rec.ID] = rec
return rec, nil
}
func (f *fakeEngine) Forget(_ context.Context, scope Scope, id RecordID) error {
rec, ok := f.records[id]
if !ok || rec.Scope != scope {
return nil
}
f.revisions[id] = append(f.revisions[id], Revision{Record: rec, Deleted: true})
delete(f.records, id)
return nil
}
func (f *fakeEngine) Revisions(_ context.Context, _ Scope, id RecordID) ([]Revision, error) {
return f.revisions[id], nil
}
// limit trims out to n when n > 0, matching the Store contract's Limit.
func limit(out []Record, n int) []Record {
if n > 0 && len(out) > n {
return out[:n]
}
return out
}
func (f *fakeEngine) Start(context.Context) error {
if f.startErr != nil {
return f.startErr
+123
View File
@@ -0,0 +1,123 @@
package memory
import "regexp"
// ThreatLevel indicates the severity of a threat detected by a Scanner.
type ThreatLevel int
const (
// ThreatNone means no suspicious patterns were found.
ThreatNone ThreatLevel = iota
// ThreatWarn means a potentially problematic pattern was found, but the
// write should be allowed with a warning logged.
ThreatWarn
// ThreatBlock means a dangerous pattern was found and the write should
// be rejected.
ThreatBlock
)
// ThreatResult describes a single threat detected during content scanning.
type ThreatResult struct {
// Level indicates the severity: none, warn, or block.
Level ThreatLevel
// Pattern is the regexp that matched (for audit logging).
Pattern string
// Message is a human-readable description of the threat.
Message string
}
// Scanner inspects memory content for dangerous or sensitive patterns before
// the content is persisted. Implementations are pluggable; the zero-value
// DefaultScanner is a reasonable default for production use.
type Scanner interface {
// ScanContent examines content and returns a ThreatResult. A result
// with Level == ThreatNone means the content is clean.
ScanContent(content string) ThreatResult
}
// DefaultScanner is the built-in Scanner implementation. It checks for
// prompt-injection patterns (block-level) and sensitive-data leakage
// (warn-level). The zero value is usable.
type DefaultScanner struct{}
// Compile-time check: DefaultScanner implements Scanner.
var _ Scanner = (*DefaultScanner)(nil)
// ── Patterns ────────────────────────────────────────────────────────────
//
// These patterns are compiled once at init time. They are intentionally
// broad to catch common variants without being so broad that they
// produce false positives on normal memory content.
var (
promptInjectionPatterns = []*regexp.Regexp{
// Override / ignore previous instructions
regexp.MustCompile(`(?i)(ignore|forget|override|disregard)\s+(all\s+)?(the\s+)?(previous|above|prior|earlier)\s+(instructions?|directions?|commands?|prompts?)`),
// Role-change / identity attacks
regexp.MustCompile(`(?i)(you\s+(are|now)\s+|act\s+as\s+|pretend\s+(to\s+be|you\s+are)\s+|roleplay\s+as\s+)(now\s+)?(an?\s+)?(different|new|another|unrestricted|uncensored)`),
// Delimiter injection (common in adversarial prompts)
regexp.MustCompile(`(?i)<\|im_start\|>`),
regexp.MustCompile(`(?i)<\|im_end\|>`),
// System prompt extraction attempts
regexp.MustCompile(`(?i)(print|output|repeat|display|show|tell\s+me)\s+(your\s+)?(system\s+)?(prompt|instructions?|directions?)\s*(exactly|verbatim)?`),
}
sensitiveDataPatterns = []*regexp.Regexp{
// API keys and secrets with common key=value patterns
regexp.MustCompile(`(?i)(api[_\s-]?key|apikey|api[_\s-]?secret)\s*[:=]\s*['"\x60]?[\w-]{8,}['"\x60]?`),
// Passwords in key=value form
regexp.MustCompile(`(?i)(password|passwd|pwd)\s*[:=]\s*['"\x60]?\S{4,}['"\x60]?`),
// Generic secrets and tokens
regexp.MustCompile(`(?i)(secret|token|credential|authorization)\s*[:=]\s*['"\x60]?[\w-]{8,}['"\x60]?`),
// Private key blocks (PEM format)
regexp.MustCompile(`(?i)-----BEGIN\s+(RSA\s+|EC\s+|DSA\s+|OPENSSH\s+)?PRIVATE\s+KEY-----`),
// JWT tokens (base64-encoded header.payload.signature pattern)
regexp.MustCompile(`eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}`),
}
)
// ScanContent examines content against the prompt-injection and
// sensitive-data patterns. Prompt-injection matches produce ThreatBlock;
// sensitive-data matches produce ThreatWarn. The first match wins.
func (s *DefaultScanner) ScanContent(content string) ThreatResult {
for _, p := range promptInjectionPatterns {
if m := p.FindString(content); m != "" {
return ThreatResult{
Level: ThreatBlock,
Pattern: p.String(),
Message: "content matches prompt injection pattern: " + truncateMatch(m),
}
}
}
for _, p := range sensitiveDataPatterns {
if m := p.FindString(content); m != "" {
return ThreatResult{
Level: ThreatWarn,
Pattern: p.String(),
Message: "content may contain sensitive data: " + truncateMatch(m),
}
}
}
return ThreatResult{Level: ThreatNone}
}
// truncateMatch limits the matched string for log/error messages so that
// a long match (e.g. an entire private key) is not echoed in full.
func truncateMatch(s string) string {
const maxMatchLen = 40
if len(s) <= maxMatchLen {
return s
}
return s[:maxMatchLen] + "..."
}
+429 -155
View File
@@ -1,6 +1,8 @@
// Package memory implements domain/memory.MemoryEngine against concrete
// backends. See docs/architecture/migration-decisions.md#5-memory-placement-and-storage
// for the record/identity model this package's builtin engine follows.
// Package memory implements domain/memory.MemoryEngine over concrete
// backends. The builtin engine persists each scope's records as markdown
// under its root -- human-readable and hand-editable -- with a per-scope
// HISTORY.md retaining the states that are no longer live. See
// docs/prds/memory-engine-unification.md for the decision this implements.
package memory
import (
@@ -9,56 +11,71 @@ import (
"encoding/hex"
"errors"
"fmt"
"log/slog"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"time"
"github.com/google/uuid"
domainmemory "github.com/samcharles93/archie-core/internal/domain/memory"
"github.com/samcharles93/archie-core/internal/memory/builtin"
"github.com/samcharles93/archie-core/internal/infrastructure/memory/builtin"
)
// EngineName is the name BuiltinEngine registers under and the default the
// config surface (archie-core-1786637490043) resolves to when no engine is
// configured.
// config surface resolves to when no engine is configured.
const EngineName = "builtin"
// markerPrefix opens the leading line every block written through this
// engine carries, embedding the record's stable ID. See the migration
// decision this package implements: an ID assigned once at Write, never
// derived from content, so a later edit elsewhere in the file can never
// orphan a Forget.
const (
markerPrefix = "<!--mem:"
markerSuffix = "-->"
)
// defaultSectionName is the section an Observation with an empty Kind is
// written to.
// defaultSectionName is the section a record with an empty Kind is written
// to.
const defaultSectionName = "general"
// BuiltinEngine implements domain/memory.MemoryEngine over one
// internal/memory/builtin.Store per identity. internal/memory/builtin is
// unchanged by this package -- see the migration decision this
// implements for why.
// liveFileName is the one live markdown document per scope, holding that
// scope's heads.
const liveFileName = "OBSERVATIONS.md"
// defaultQueryLimit bounds Query when the caller names no limit: 0 in
// Query.Limit means "the engine's default", never unlimited, and List is the
// unbounded read for one scope.
const defaultQueryLimit = 100
// contentScanner is the family's content scanner, applied in Create and
// Update. Those two are the one choke point every producer crosses, so the
// curator's model-extracted content is covered as well as the chat tool's
// (docs/prds/memory-engine-unification.md §7).
var contentScanner domainmemory.Scanner = &domainmemory.DefaultScanner{}
// scopeStores is one scope's on-disk pair: the heads, and the states that
// used to be heads.
type scopeStores struct {
live *builtin.Store
history *builtin.Store
}
// BuiltinEngine implements domain/memory.MemoryEngine over one pair of
// builtin.Store documents per scope.
type BuiltinEngine struct {
root string
maxFileBytes int
mu sync.Mutex
stores map[string]*builtin.Store // keyed by identity, lazily created
stores map[string]*scopeStores // keyed by scope key, lazily created
clock domainmemory.Clock
log *slog.Logger
}
// NewBuiltinEngine builds an engine that persists each identity's
// observations under its own subdirectory of root, named
// hex(sha256(identity)) -- never the identity string itself, so an
// arbitrary identity can never traverse outside root and two identities
// differing only in filesystem-equivalent characters can never collide.
// maxFileBytes <= 0 uses the store's own default.
// NewBuiltinEngine builds an engine that persists each scope under its own
// subdirectory of root, named hex(sha256(scope.Key())): never the key or an
// id as a path component, so an opaque channel id can never traverse out of
// root and two ids a filesystem treats as equivalent can never collide.
// maxFileBytes <= 0 uses the store's own default for the live document;
// HISTORY.md always uses historyMaxFileBytes.
func NewBuiltinEngine(root string, maxFileBytes int) *BuiltinEngine {
return &BuiltinEngine{root: root, maxFileBytes: maxFileBytes, stores: make(map[string]*builtin.Store)}
return &BuiltinEngine{root: root, maxFileBytes: maxFileBytes, stores: make(map[string]*scopeStores)}
}
func (e *BuiltinEngine) Name() string { return EngineName }
@@ -68,7 +85,21 @@ func (e *BuiltinEngine) Manifest() domainmemory.Manifest {
return domainmemory.Manifest{RequiresNetwork: false}
}
func (e *BuiltinEngine) Bind(domainmemory.Registrar) {}
// Bind takes the registrar's clock, which stamps CreatedAt/UpdatedAt, and its
// optional logger.
//
// It deliberately ignores Registrar.Events. The only warn-level finding this
// engine produces is a scanner hit, and there is no ratified memory event
// vocabulary to emit it in: internal/events declares kinds for tasks,
// curators, workflow stages and scheduling, none for memory, and the
// composition binds no memory event sink at all (bootstrap registers the
// engine with an empty Registrar). Emitting an invented kind into a sink
// nobody binds would look wired and not be. The scanner's warning is a
// diagnostic, not an event, and goes to Registrar.Log (see scanContent).
func (e *BuiltinEngine) Bind(host domainmemory.Registrar) {
e.clock = host.Clock
e.log = host.Log
}
func (e *BuiltinEngine) Start(context.Context) error { return nil }
@@ -78,37 +109,77 @@ func (e *BuiltinEngine) Health(context.Context) domainmemory.Health {
func (e *BuiltinEngine) Stop(context.Context) error { return nil }
// storeFor returns the identity's store, creating and caching it on first
// use.
func (e *BuiltinEngine) storeFor(identity string) (*builtin.Store, error) {
e.mu.Lock()
defer e.mu.Unlock()
if s, ok := e.stores[identity]; ok {
return s, nil
// systemClock is the fallback for an engine no registry bound. Registrar's
// own doc says a nil Clock is replaced at construction, but a directly
// constructed engine (a test, an embedder) has no registry at all, and a
// clock nil-panic on a write would be a poor way to find that out.
type systemClock struct{}
func (systemClock) Now() time.Time { return time.Now() }
var fallbackClock domainmemory.Clock = systemClock{}
func (e *BuiltinEngine) now() time.Time {
if e.clock == nil {
return fallbackClock.Now()
}
dir := filepath.Join(e.root, identityDir(identity))
if err := os.MkdirAll(dir, 0o755); err != nil {
return nil, fmt.Errorf("memory: builtin engine: identity directory %s unusable: %w", dir, err)
}
s, err := builtin.NewStore(filepath.Join(dir, "OBSERVATIONS.md"), e.maxFileBytes)
if err != nil {
return nil, err
}
e.stores[identity] = s
return s, nil
return e.clock.Now()
}
// identityDir maps an identity to a filesystem-safe, collision-resistant
// ── storage ────────────────────────────────────────────────────────────
// scopeStoresFor returns the scope's document pair, opening it on first use
// and caching it. create is false on every read path: a read must not leave
// a scope directory behind, and a scope nothing was ever written to answers
// as absent rather than as empty-but-created.
//
// Callers must have validated the scope already: Scope.Key() is empty for an
// invalid scope, and caching on "" would pool every invalid scope into one
// store under one directory.
func (e *BuiltinEngine) scopeStoresFor(scope domainmemory.Scope, create bool) (*scopeStores, bool, error) {
key := scope.Key()
e.mu.Lock()
defer e.mu.Unlock()
if s, ok := e.stores[key]; ok {
return s, true, nil
}
dir := filepath.Join(e.root, scopeDirName(scope))
if create {
if err := os.MkdirAll(dir, 0o755); err != nil {
return nil, false, fmt.Errorf("memory: builtin engine: scope directory %s unusable: %w", dir, err)
}
} else if _, err := os.Stat(dir); err != nil {
if os.IsNotExist(err) {
return nil, false, nil
}
return nil, false, fmt.Errorf("memory: builtin engine: scope directory %s unusable: %w", dir, err)
}
live, err := builtin.NewStore(filepath.Join(dir, liveFileName), e.maxFileBytes)
if err != nil {
return nil, false, err
}
history, err := builtin.NewStore(filepath.Join(dir, historyFileName), historyMaxFileBytes)
if err != nil {
return nil, false, err
}
stores := &scopeStores{live: live, history: history}
e.stores[key] = stores
return stores, true, nil
}
// scopeDirName maps a scope to a filesystem-safe, collision-resistant
// directory name.
func identityDir(identity string) string {
sum := sha256.Sum256([]byte(identity))
func scopeDirName(scope domainmemory.Scope) string {
sum := sha256.Sum256([]byte(scope.Key()))
return hex.EncodeToString(sum[:])
}
// sectionFor returns the section an Observation's Kind maps to, and
// validates it the same way builtin already validates any section name --
// so an invalid Kind fails with the same error shape memory_edit already
// returns for an invalid section, not a new one.
// sectionFor returns the section a record's Kind maps to, validated the same
// way builtin validates any section name -- so an invalid Kind fails with the
// error shape the memory tool already returns for an invalid section, not a
// new one.
func sectionFor(kind string) (string, error) {
if kind == "" {
kind = defaultSectionName
@@ -119,111 +190,314 @@ func sectionFor(kind string) (string, error) {
return kind, nil
}
func (e *BuiltinEngine) Write(_ context.Context, obs domainmemory.Observation) (domainmemory.Record, error) {
if err := obs.Validate(); err != nil {
return domainmemory.Record{}, err
}
section, err := sectionFor(obs.Kind)
if err != nil {
return domainmemory.Record{}, err
}
s, err := e.storeFor(obs.Identity)
if err != nil {
return domainmemory.Record{}, err
}
id := uuid.NewString()
marked := markerPrefix + id + markerSuffix + "\n" + obs.Content
if _, err := s.Add(section, marked); err != nil {
return domainmemory.Record{}, err
}
return domainmemory.Record{
ID: id,
Identity: obs.Identity,
Kind: section,
Content: obs.Content,
At: obs.At,
Metadata: obs.Metadata,
}, nil
}
func (e *BuiltinEngine) Query(ctx context.Context, q domainmemory.Query) ([]domainmemory.Record, error) {
if err := q.Validate(); err != nil {
return nil, err
}
all, err := e.List(ctx, q.Identity)
if err != nil {
return nil, err
}
if q.Text == "" {
return limitRecords(all, q.Limit), nil
}
var matched []domainmemory.Record
for _, r := range all {
if strings.Contains(r.Content, q.Text) {
matched = append(matched, r)
// scanContent applies the family's scanner to content about to be persisted,
// before anything is written.
//
// A block-level threat fails the write loudly -- loudly, because a scanner
// hit that only logs is a control that looks wired and is not -- while a
// warn-level hit is stored and logged, which is what its level is documented
// to mean ("allow the write but emit a warning"). Warn covers the
// sensitive-data patterns: an API key, a token or a private key in stored
// content is worth an audit line even though it is not worth refusing the
// write over, and a warning that goes nowhere is the same as no scan at all.
func (e *BuiltinEngine) scanContent(content string) error {
result := contentScanner.ScanContent(content)
switch result.Level {
case domainmemory.ThreatBlock:
return fmt.Errorf("memory: builtin engine: refusing to persist content: %s", result.Message)
case domainmemory.ThreatWarn:
if e.log != nil {
e.log.Warn("memory: stored content matched a sensitive-data pattern",
"pattern", result.Pattern, "reason", result.Message)
}
}
return limitRecords(matched, q.Limit), nil
}
func limitRecords(records []domainmemory.Record, limit int) []domainmemory.Record {
if limit > 0 && len(records) > limit {
return records[:limit]
}
return records
}
func (e *BuiltinEngine) List(_ context.Context, identity string) ([]domainmemory.Record, error) {
if identity == "" {
return nil, errors.New("memory: builtin engine: identity must not be empty")
}
s, err := e.storeFor(identity)
if err != nil {
return nil, err
}
return parseRecords(identity, s.Render()), nil
}
func (e *BuiltinEngine) Forget(_ context.Context, id string) error {
if id == "" {
return errors.New("memory: builtin engine: id must not be empty")
}
marker := markerPrefix + id + markerSuffix
e.mu.Lock()
stores := make([]*builtin.Store, 0, len(e.stores))
for _, s := range e.stores {
stores = append(stores, s)
}
e.mu.Unlock()
// Forgetting an id that does not exist is not an error (contract.go's
// Forget doc): the end state -- id absent -- already holds whether or
// not any store ever had it, so checking every known identity's store
// and returning nil regardless of a miss is correct, not a fallback
// masking a real failure.
for _, s := range stores {
section, ok := sectionOf(s.Render(), id)
if !ok {
continue
}
if _, err := s.Remove(section, marker, true); err != nil {
return err
}
return nil
}
return nil
}
// sectionOf finds which section of rendered carries a block whose marker
// names id, so Forget can call Store.Remove with the section it actually
// requires without a mutating probe. ok is false if no block does.
func sectionOf(rendered, id string) (section string, ok bool) {
for _, rec := range parseRecords("", rendered) {
if rec.ID == id {
return rec.Kind, true
// storeContent returns the content a block will actually hold, so that the
// record Create and Update answer with is the record a later Get reads back
// rather than what the caller handed in.
//
// The trimming is what makes that true at the content's edges: builtin.Store
// trims a block, so content that begins or ends with whitespace would come
// back without it -- or, for whitespace that is itself a blank line, as an
// empty record. Content that is nothing but whitespace is rejected here for
// the same reason. Interior blank lines and "## " lines are the block
// format's own delimiters rather than whitespace at an edge, and are carried
// through instead: renderBlock escapes them and parseBlocks undoes it, so
// accepted content is never cut at one.
func storeContent(content string) (string, error) {
trimmed := strings.TrimSpace(content)
if trimmed == "" {
return "", errors.New("memory: builtin engine: content must not be empty")
}
return trimmed, nil
}
// findBlock returns the live block addressed by id.
func findBlock(store *builtin.Store, id domainmemory.RecordID) (parsedBlock, bool) {
for _, block := range parseBlocks(store.Render()) {
if block.marker.ID == string(id) {
return block, true
}
}
return "", false
return parsedBlock{}, false
}
func notFound(scope domainmemory.Scope, id domainmemory.RecordID) error {
return fmt.Errorf("memory: builtin engine: %w: id %q in scope %s", domainmemory.ErrNotFound, id, scope)
}
// ── Store ──────────────────────────────────────────────────────────────
// Create records one new memory: the id is assigned once, here, and is the
// only address the record ever has.
func (e *BuiltinEngine) Create(_ context.Context, in domainmemory.NewRecord) (domainmemory.Record, error) {
if err := in.Validate(); err != nil {
return domainmemory.Record{}, err
}
section, err := sectionFor(in.Kind)
if err != nil {
return domainmemory.Record{}, err
}
if err := e.scanContent(in.Content); err != nil {
return domainmemory.Record{}, err
}
content, err := storeContent(in.Content)
if err != nil {
return domainmemory.Record{}, err
}
stores, _, err := e.scopeStoresFor(in.Scope, true)
if err != nil {
return domainmemory.Record{}, err
}
now := e.now()
marker := markerData{
ID: uuid.NewString(),
Revision: 1,
Kind: section,
Author: in.Author,
OriginUser: string(in.OriginUser),
Source: in.Source,
CreatedAt: now,
UpdatedAt: now,
}
block, err := renderBlock(marker, content)
if err != nil {
return domainmemory.Record{}, err
}
if _, err := stores.live.Add(section, block); err != nil {
return domainmemory.Record{}, err
}
return recordOf(marker, in.Scope, content), nil
}
func (e *BuiltinEngine) Get(_ context.Context, scope domainmemory.Scope, id domainmemory.RecordID) (domainmemory.Record, error) {
if err := scope.Validate(); err != nil {
return domainmemory.Record{}, err
}
if id == "" {
return domainmemory.Record{}, errors.New("memory: builtin engine: id must not be empty")
}
stores, ok, err := e.scopeStoresFor(scope, false)
if err != nil {
return domainmemory.Record{}, err
}
if !ok {
return domainmemory.Record{}, notFound(scope, id)
}
block, ok := findBlock(stores.live, id)
if !ok {
return domainmemory.Record{}, notFound(scope, id)
}
return block.record(scope), nil
}
// List returns one scope's heads, most recently updated first.
func (e *BuiltinEngine) List(_ context.Context, scope domainmemory.Scope) ([]domainmemory.Record, error) {
if err := scope.Validate(); err != nil {
return nil, err
}
stores, ok, err := e.scopeStoresFor(scope, false)
if err != nil {
return nil, err
}
if !ok {
return nil, nil
}
blocks := parseBlocks(stores.live.Render())
records := make([]domainmemory.Record, 0, len(blocks))
for _, block := range blocks {
records = append(records, block.record(scope))
}
return orderHeads(records), nil
}
// orderHeads returns heads most recently updated first. The input is in
// document order, which is creation order for a document this engine wrote;
// reversing before the stable sort makes creation order the tie-break for two
// records sharing an UpdatedAt, which is the common case for a coarse clock or
// one Create burst.
func orderHeads(records []domainmemory.Record) []domainmemory.Record {
slices.Reverse(records)
slices.SortStableFunc(records, func(a, b domainmemory.Record) int {
return b.UpdatedAt.Compare(a.UpdatedAt)
})
return records
}
// Query returns the heads of exactly the scopes q names -- the caller's read
// set, never widened here -- grouped in the order those scopes were given and
// bounded by q.Limit across the whole result.
func (e *BuiltinEngine) Query(ctx context.Context, q domainmemory.Query) ([]domainmemory.Record, error) {
if err := q.Validate(); err != nil {
return nil, err
}
limit := q.Limit
if limit == 0 {
limit = defaultQueryLimit
}
var out []domainmemory.Record
for _, scope := range q.Scopes {
heads, err := e.List(ctx, scope)
if err != nil {
return nil, err
}
for _, record := range heads {
if q.Text != "" && !strings.Contains(record.Content, q.Text) {
continue
}
out = append(out, record)
if len(out) == limit {
return out, nil
}
}
}
return out, nil
}
// Update supersedes a record's content in place and returns the new state.
//
// A record may not move between scopes, which is why RecordUpdate carries no
// scope to change: it is addressed by the scope it is already in.
func (e *BuiltinEngine) Update(_ context.Context, in domainmemory.RecordUpdate) (domainmemory.Record, error) {
if err := in.Validate(); err != nil {
return domainmemory.Record{}, err
}
// Scanned before anything is written: a refused update must leave both
// the record and its history untouched.
if err := e.scanContent(in.Content); err != nil {
return domainmemory.Record{}, err
}
content, err := storeContent(in.Content)
if err != nil {
return domainmemory.Record{}, err
}
stores, ok, err := e.scopeStoresFor(in.Scope, false)
if err != nil {
return domainmemory.Record{}, err
}
if !ok {
return domainmemory.Record{}, notFound(in.Scope, in.ID)
}
// Read the document as it is on disk before anything is decided from it.
// This engine caches the scope's document and rewrites it whole (the PRD
// leaves "two processes, one scope file" undecided), so the copy it holds
// can be a whole write out of date -- the Expected check below would
// compare against a revision that is already superseded, and the rewrite
// that follows it would write the stale copy back over the current one.
// What this does not fix: the rewrite is still last-writer-wins for this
// record and every other one, between the reload and the write.
if err := stores.live.Reload(); err != nil {
return domainmemory.Record{}, err
}
current, ok := findBlock(stores.live, in.ID)
if !ok {
return domainmemory.Record{}, notFound(in.Scope, in.ID)
}
// Expected is what makes a lost update detectable: the revision the
// caller last saw is the check, against the revision just read.
if in.Expected != 0 && in.Expected != current.marker.Revision {
return domainmemory.Record{}, fmt.Errorf(
"memory: builtin engine: %w: record %q is at revision %d, not %d",
domainmemory.ErrStaleRevision, in.ID, current.marker.Revision, in.Expected,
)
}
now := e.now()
next := current.marker
next.Revision = current.marker.Revision + 1
next.Author = in.Author
next.Source = in.Source
next.UpdatedAt = now
// Crash safety, in this order: the superseded state is durable in
// HISTORY.md before the live block is touched. A crash between the two
// leaves an extra history entry and an unchanged record -- recoverable,
// per §2 -- and never a record whose previous state is gone.
if err := appendHistory(stores.history, current.marker, current.content(), now, false); err != nil {
return domainmemory.Record{}, err
}
block, err := renderBlock(next, content)
if err != nil {
return domainmemory.Record{}, err
}
if _, err := stores.live.Replace(current.section, current.text, block, true); err != nil {
return domainmemory.Record{}, err
}
return recordOf(next, in.Scope, content), nil
}
// Forget removes a record's live state, retaining a deleted revision so its
// provenance outlives its content. Forgetting an id that is not there is not
// an error: the end state, id absent, already holds.
func (e *BuiltinEngine) Forget(_ context.Context, scope domainmemory.Scope, id domainmemory.RecordID) error {
if err := scope.Validate(); err != nil {
return err
}
if id == "" {
return errors.New("memory: builtin engine: id must not be empty")
}
stores, ok, err := e.scopeStoresFor(scope, false)
if err != nil {
return err
}
if !ok {
return nil
}
current, ok := findBlock(stores.live, id)
if !ok {
return nil
}
// The same crash-safe order as Update: the deleted revision is durable
// before the live block disappears, so the record's provenance is never
// the part that is lost.
if err := appendHistory(stores.history, current.marker, current.content(), e.now(), true); err != nil {
return err
}
_, err = stores.live.Remove(current.section, current.text, true)
return err
}
// Revisions returns a record's retained states, oldest first, including the
// deleted revision Forget records. A record that never existed is empty, not
// an error.
func (e *BuiltinEngine) Revisions(_ context.Context, scope domainmemory.Scope, id domainmemory.RecordID) ([]domainmemory.Revision, error) {
if err := scope.Validate(); err != nil {
return nil, err
}
if id == "" {
return nil, errors.New("memory: builtin engine: id must not be empty")
}
stores, ok, err := e.scopeStoresFor(scope, false)
if err != nil {
return nil, err
}
if !ok {
return nil, nil
}
return revisionsOf(stores.history, scope, id), nil
}
File diff suppressed because it is too large. Load diff
+76
View File
@@ -0,0 +1,76 @@
package memory
import (
"fmt"
"time"
domainmemory "github.com/samcharles93/archie-core/internal/domain/memory"
"github.com/samcharles93/archie-core/internal/infrastructure/memory/builtin"
)
// historyFileName is the second document in each scope's directory: every
// state the scope's records no longer live in -- superseded by an Update, or
// removed by a Forget -- so a record's provenance outlives its content
// (docs/prds/memory-engine-unification.md §2).
const historyFileName = "HISTORY.md"
// historySectionName is the single section every retained state is appended
// to. One section is enough: each block's marker carries its record id, so
// grouping by kind would be a second index over the same file.
const historySectionName = "history"
// historyMaxFileBytes bounds each scope's HISTORY.md, deliberately far above
// the live document's bound (builtin's 100KB default, which is what
// production passes: bootstrap builds the engine with a 0 maxFileBytes).
//
// It has its own bound because history grows on every Update and every
// Forget while the live document does not. Inheriting the live document's
// bound would make a scope start refusing updates after a handful of
// revisions of one large record -- trading the PRD's open "unbounded
// history" question ("Not determined", no compaction policy) for a new
// failure mode at the point of write. That question stays open; this bound
// only makes reaching it a deliberate, enormous amount of history rather
// than an accident.
const historyMaxFileBytes = 4 * 1024 * 1024
// appendHistory appends one retained state -- the record as it was while
// live, plus when it stopped being live and whether it was deleted -- to the
// scope's history document. Content is passed beside m because the marker
// carries provenance, never the record's bytes.
func appendHistory(h *builtin.Store, m markerData, content string, at time.Time, deleted bool) error {
m.SupersededAt = &at
m.Deleted = deleted
block, err := renderBlock(m, content)
if err != nil {
return err
}
if _, err := h.Add(historySectionName, block); err != nil {
return fmt.Errorf("memory: builtin engine: retaining the superseded state of record %q: %w", m.ID, err)
}
return nil
}
// revisionsOf returns one record's retained states as HISTORY.md holds them,
// oldest first. Document order is append order, and a retained state is
// appended exactly when it stops being live, so no sort is needed -- or
// wanted: a crash between a history append and the live write leaves two
// entries for one revision (§2, recoverable), and sorting would not make
// that less true.
func revisionsOf(h *builtin.Store, scope domainmemory.Scope, id domainmemory.RecordID) []domainmemory.Revision {
var revisions []domainmemory.Revision
for _, block := range parseBlocks(h.Render()) {
if block.marker.ID != string(id) {
continue
}
var supersededAt time.Time
if block.marker.SupersededAt != nil {
supersededAt = *block.marker.SupersededAt
}
revisions = append(revisions, domainmemory.Revision{
Record: block.record(scope),
SupersededAt: supersededAt,
Deleted: block.marker.Deleted,
})
}
return revisions
}
+82
View File
@@ -0,0 +1,82 @@
package memory
import (
"bytes"
"encoding/json"
"fmt"
"strings"
"time"
)
// The engine writes each record as one blank-line-delimited markdown block
// whose first line is a marker comment followed by the record's content, so
// a block stays readable and editable by hand. The marker began life as a
// bare id (`<!--mem:<ulid>-->`, the incumbent format); it is JSON now
// because the retained provenance -- revision, kind, author, originating
// user, source and both timestamps -- has nowhere else to live, which is
// exactly what the PRD's Settles table records the bare form could not
// carry.
const (
markerPrefix = "<!--mem:"
markerSuffix = "-->"
)
// markerData is the provenance one block's marker carries. The same shape is
// written into a scope's live document and into its HISTORY.md; the last two
// fields are set only on a retained state, so a live block never carries
// them.
//
// json escapes newlines, so the marker is always exactly one line: a field
// containing one cannot break the block's first-line shape.
type markerData struct {
ID string `json:"id"`
Revision int `json:"revision"`
Kind string `json:"kind"`
Author string `json:"author,omitempty"`
OriginUser string `json:"origin_user,omitempty"`
Source string `json:"source,omitempty"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
// SupersededAt is when this state stopped being live. Set only on a
// state written to HISTORY.md.
SupersededAt *time.Time `json:"superseded_at,omitempty"`
// Deleted marks the state Forget retained: the record's last live
// content, kept so its provenance outlives it. Set only in HISTORY.md.
Deleted bool `json:"deleted,omitempty"`
}
// renderMarker encodes one marker line.
func renderMarker(m markerData) (string, error) {
var buf bytes.Buffer
enc := json.NewEncoder(&buf)
// The marker is documented as hand-editable, so it must not be
// HTML-escaped into \u003c runs an operator cannot read back. The suffix
// is still unambiguous either way: the body is cut at the line's end.
enc.SetEscapeHTML(false)
if err := enc.Encode(m); err != nil {
return "", fmt.Errorf("memory: builtin engine: encoding a record marker: %w", err)
}
return markerPrefix + strings.TrimRight(buf.String(), "\n") + markerSuffix, nil
}
// parseMarker decodes a block's first line. ok is false for a line that is
// not a marker, or one that names no record: a block without a usable marker
// is not one this engine ever wrote.
func parseMarker(line string) (markerData, bool) {
body, ok := strings.CutPrefix(line, markerPrefix)
if !ok {
return markerData{}, false
}
body, ok = strings.CutSuffix(body, markerSuffix)
if !ok {
return markerData{}, false
}
var m markerData
if err := json.Unmarshal([]byte(body), &m); err != nil {
return markerData{}, false
}
if m.ID == "" {
return markerData{}, false
}
return m, true
}
+128 -39
View File
@@ -6,31 +6,140 @@ import (
domainmemory "github.com/samcharles93/archie-core/internal/domain/memory"
)
// sectionHeaderPrefix matches internal/memory/builtin's own format (its
// sectionHeaderPrefix constant is unexported): a "## " line starts a named
// section, and blocks within it are separated by one blank line. This
// package only ever reads what it wrote through builtin.Store.Add, so it
// only needs to read that one documented format back, not the general
// case builtin.Store itself parses.
// sectionHeaderPrefix matches the markdown store's own format (builtin's
// sectionHeaderPrefix is unexported): a "## " line starts a named section,
// and blocks within it are separated by one blank line. This package only
// reads back what it wrote through builtin.Store.Add, so it only needs that
// one documented format, not the general case builtin.Store itself parses.
//
// Those two delimiters are the format's structure, and builtin.Store owns
// them: a blank line ends a block and a "## " line starts a section, so
// content holding either would be read back as structure -- cut at the
// delimiter, the remainder unaddressable. Content is therefore escaped on
// the way out and unescaped on the way back (see encodeBlockContent), which
// keeps a record's content whole without giving the store a second format to
// parse.
const sectionHeaderPrefix = "## "
// parseRecords reads every marked block out of rendered and returns it as
// a Record scoped to identity. A block with no marker was not written by
// this engine (e.g. hand-edited) and is skipped: this engine's contract is
// only ever asked about records it assigned an ID to.
func parseRecords(identity, rendered string) []domainmemory.Record {
var records []domainmemory.Record
// parsedBlock is one marked block read back out of a rendered document: the
// provenance its marker carried, the section it sat in, and its exact text
// within that document. Update replaces a block by that text, so it is never
// reconstructed from fields and can never disagree with what is on disk.
type parsedBlock struct {
marker markerData
section string
text string
}
// content returns the block's content with the marker line stripped and the
// format's escaping undone.
func (b parsedBlock) content() string {
_, rest, _ := strings.Cut(b.text, "\n")
return decodeBlockContent(rest)
}
// record renders the block as a record of scope. Kind comes from the marker
// rather than the section: every record's retained states live in one
// "history" section, so the section cannot be the source of a kind.
func (b parsedBlock) record(scope domainmemory.Scope) domainmemory.Record {
return recordOf(b.marker, scope, b.content())
}
// record renders a marker and its content as a record of scope.
func recordOf(m markerData, scope domainmemory.Scope, content string) domainmemory.Record {
return domainmemory.Record{
ID: domainmemory.RecordID(m.ID),
Scope: scope,
Kind: m.Kind,
Content: content,
Revision: m.Revision,
Author: m.Author,
OriginUser: domainmemory.IdentityID(m.OriginUser),
Source: m.Source,
CreatedAt: m.CreatedAt,
UpdatedAt: m.UpdatedAt,
}
}
// renderBlock renders one record as the markdown block the store persists:
// the marker line, then the content.
func renderBlock(m markerData, content string) (string, error) {
line, err := renderMarker(m)
if err != nil {
return "", err
}
return line + "\n" + encodeBlockContent(content), nil
}
// contentEscape is the backslash that marks a content line the format would
// otherwise read as structure.
const contentEscape = `\`
// encodeBlockContent escapes the content lines the store's format would
// otherwise read as its own structure -- a blank line ends a block, and a
// "## " line starts a section -- so that a record's content survives the
// write whole (see sectionHeaderPrefix). An empty line becomes one backslash,
// and a line already starting with a backslash or with "## " gains one, so
// the two are never ambiguous: everything is written verbatim except lines
// that start with a backslash, and every content line can therefore be
// recovered by dropping the backslash it starts with (decodeBlockContent).
// The common record -- one paragraph, no delimiter in sight -- is written
// exactly as the caller wrote it, so the document stays hand-editable.
func encodeBlockContent(content string) string {
lines := strings.Split(content, "\n")
for i, line := range lines {
switch {
case line == "":
lines[i] = contentEscape
case strings.HasPrefix(line, contentEscape), strings.HasPrefix(line, sectionHeaderPrefix):
lines[i] = contentEscape + line
}
}
return strings.Join(lines, "\n")
}
// decodeBlockContent is encodeBlockContent's inverse: it returns the content
// the block's lines were rendered from.
func decodeBlockContent(text string) string {
lines := strings.Split(text, "\n")
for i, line := range lines {
if line == contentEscape {
lines[i] = ""
continue
}
lines[i] = strings.TrimPrefix(line, contentEscape)
}
return strings.Join(lines, "\n")
}
// parseBlocks reads every marked block out of a rendered document.
//
// A block with no marker, or one whose marker does not decode, was not
// written by this engine -- an operator hand-edited the file, or it predates
// the marker -- and is skipped. This engine's Store contract is only ever
// asked about records it assigned an id to, so inventing an id for an
// unmarked block would be a worse answer than not seeing it.
//
// Blocks are returned in document order, which is append order for a
// document this engine wrote, and therefore the order HISTORY.md's retained
// states come back in.
func parseBlocks(rendered string) []parsedBlock {
var blocks []parsedBlock
section := ""
var blockLines []string
var lines []string
flush := func() {
if len(blockLines) == 0 {
if len(lines) == 0 {
return
}
if rec, ok := parseBlock(identity, section, strings.Join(blockLines, "\n")); ok {
records = append(records, rec)
text := strings.Join(lines, "\n")
lines = nil
first, _, _ := strings.Cut(text, "\n")
marker, ok := parseMarker(first)
if !ok {
return
}
blockLines = nil
blocks = append(blocks, parsedBlock{marker: marker, section: section, text: text})
}
for line := range strings.SplitSeq(rendered, "\n") {
@@ -43,28 +152,8 @@ func parseRecords(identity, rendered string) []domainmemory.Record {
flush()
continue
}
blockLines = append(blockLines, line)
lines = append(lines, line)
}
flush()
return records
}
// parseBlock splits one block back into its marker-carried ID and content.
// ok is false for a block with no recognizable marker.
func parseBlock(identity, section, block string) (domainmemory.Record, bool) {
first, rest, _ := strings.Cut(block, "\n")
id, ok := strings.CutPrefix(first, markerPrefix)
if !ok {
return domainmemory.Record{}, false
}
id, ok = strings.CutSuffix(id, markerSuffix)
if !ok {
return domainmemory.Record{}, false
}
return domainmemory.Record{
ID: id,
Identity: identity,
Kind: section,
Content: rest,
}, true
return blocks
}
@@ -18,6 +18,14 @@ const Name = "session-memory"
// written.
const ActionExtracted = "memory.extracted"
// ActionSkipped records a session the pass could not attribute to exactly
// one participant, so nothing was written for it. The reason names how many
// distinct senders were found (zero for a dashboard or webhook session,
// more than one for a group chat), and how many user messages carried no
// sender at all -- unknown provenance rules a session out on its own, beside
// any number of identified senders.
const ActionSkipped = "memory.skipped"
// DefaultInterval is the check-in cadence used when nothing more
// specific configures one, matching skillcurator.DefaultInterval's
// convention (no per-curator interval config surface exists yet).
@@ -103,7 +111,7 @@ func (c *Curator) Pass(ctx context.Context, in curator.PassInput) (curator.PassR
var actions []curator.Action
for _, sess := range sessions {
action, err := c.reviewOne(ctx, engine, sess.ID)
action, err := c.reviewOne(ctx, engine, sess)
if err != nil {
return curator.PassResult{}, fmt.Errorf("session %q: %w", sess.ID, err)
}
@@ -126,11 +134,14 @@ func effectiveSince(since, now time.Time) time.Time {
}
// reviewOne extracts observations from one session's recent messages and
// writes them through engine. Returns nil (no Action) when nothing was
// extracted -- matching the skill curator's "a pass with nothing to
// report is not an error."
func (c *Curator) reviewOne(ctx context.Context, engine domainmemory.MemoryEngine, sessionID string) (*curator.Action, error) {
msgs, err := c.host.Conversations.Messages(ctx, sessionID, messageTailSize)
// writes them through engine as agent-user memory for the session's single
// participant. Returns nil (no Action) when nothing was extracted --
// matching the skill curator's "a pass with nothing to report is not an
// error" -- or when the session carries no messages at all. A session that
// cannot be attributed to one participant is skipped with an Action saying
// so.
func (c *Curator) reviewOne(ctx context.Context, engine domainmemory.MemoryEngine, sess curator.SessionSummary) (*curator.Action, error) {
msgs, err := c.host.Conversations.Messages(ctx, sess.ID, messageTailSize)
if err != nil {
return nil, err
}
@@ -138,6 +149,20 @@ func (c *Curator) reviewOne(ctx context.Context, engine domainmemory.MemoryEngin
return nil, nil
}
// Resolve the participant before spending a model call: an
// unattributable session has nowhere addressable to write, so there is
// nothing a model response could add. ScopeAgentUser is addressed by
// both ids, and a guess would file one person's facts under another's.
participant, reason := sessionParticipant(msgs)
if participant == "" {
return &curator.Action{
At: c.host.Clock.Now(),
Type: ActionSkipped,
Detail: sess.ID,
Reason: reason,
}, nil
}
text, err := c.askModel(ctx, msgs)
if err != nil {
// A real model-call failure (provider down, timeout) is a Pass
@@ -159,12 +184,25 @@ func (c *Curator) reviewOne(ctx context.Context, engine domainmemory.MemoryEngin
facts = facts[:maxObservationsPerSession]
}
// Agent-user is the only defensible home for a derived fact about a
// person: agent scope would leak user A's facts to user B through the
// same agent, user scope would pool them across agents, and the session
// id this used to write stops meaning anything when the session ends.
// ScopeAgentUser is also what a later chat turn reads back, which is
// the whole point of writing these at all.
scope := domainmemory.Scope{
Kind: domainmemory.ScopeAgentUser,
Agent: domainmemory.AgentID(sess.AgentID),
User: domainmemory.IdentityID(participant),
}
for _, fact := range facts {
if _, err := engine.Write(ctx, domainmemory.Observation{
Identity: sessionID,
Kind: "note",
Content: fact,
At: c.host.Clock.Now(),
if _, err := engine.Create(ctx, domainmemory.NewRecord{
Scope: scope,
Kind: "note",
Content: fact,
Author: Name,
OriginUser: domainmemory.IdentityID(participant),
Source: sess.ID,
}); err != nil {
return nil, err
}
@@ -173,11 +211,48 @@ func (c *Curator) reviewOne(ctx context.Context, engine domainmemory.MemoryEngin
return &curator.Action{
At: c.host.Clock.Now(),
Type: ActionExtracted,
Detail: fmt.Sprintf("%s: wrote %d observation(s)", sessionID, len(facts)),
Detail: fmt.Sprintf("%s: wrote %d observation(s)", sess.ID, len(facts)),
Reason: "session active since last pass",
}, nil
}
// sessionParticipant returns the one participant a session's memory may be
// attributed to, or -- when there is none -- the reason there isn't.
//
// Only user-role messages carry a person: an assistant message carries the
// bot, so it can never manufacture a participant.
//
// A user-role message with an empty SenderID is part of the excerpt the
// model reads but names nobody, so the session is unattributable outright,
// even beside exactly one identified sender. The alternative -- skipping it
// and attributing the rest -- would credit whoever that sender is with
// facts drawn from content of unknown provenance, which is the isolation
// failure the agent-user scope exists to prevent.
func sessionParticipant(msgs []curator.ConversationMessage) (participant, reason string) {
senders := make(map[string]struct{})
unidentified := 0
for _, m := range msgs {
if m.Role != "user" {
continue
}
if m.SenderID == "" {
unidentified++
continue
}
senders[m.SenderID] = struct{}{}
}
if unidentified > 0 {
return "", fmt.Sprintf("no single participant: %d distinct sender(s), %d unidentified user message(s)", len(senders), unidentified)
}
if len(senders) == 1 {
for id := range senders {
return id, ""
}
}
return "", fmt.Sprintf("no single participant: %d distinct sender(s)", len(senders))
}
// extractPrompt instructs the model to return a JSON array of short,
// durable strings -- facts, preferences, or corrections worth keeping --
// or [] if nothing in the excerpt is worth keeping. No prose outside the
@@ -3,6 +3,7 @@ package sessioncurator
import (
"context"
"errors"
"fmt"
"testing"
"time"
@@ -14,6 +15,7 @@ import (
type fakeConversations struct {
sessions map[string]time.Time // id -> last active
agents map[string]string // id -> the agent that served it
messages map[string][]curator.ConversationMessage
}
@@ -23,7 +25,7 @@ func (f *fakeConversations) RecentSessions(_ context.Context, since time.Time) (
if at.Before(since) {
continue
}
out = append(out, curator.SessionSummary{ID: id, LastActive: at})
out = append(out, curator.SessionSummary{ID: id, AgentID: f.agents[id], LastActive: at})
}
return out, nil
}
@@ -36,6 +38,17 @@ func (f *fakeConversations) Messages(_ context.Context, sessionID string, n int)
return msgs, nil
}
// newFakeConversations builds the single-session fixture every write-path
// case starts from: session "s1", served by agent "winter", with the given
// message tail.
func newFakeConversations(msgs ...curator.ConversationMessage) *fakeConversations {
return &fakeConversations{
sessions: map[string]time.Time{"s1": time.Unix(1000, 0)},
agents: map[string]string{"s1": "winter"},
messages: map[string][]curator.ConversationMessage{"s1": msgs},
}
}
// fakeLLM returns a canned response per call, or the same response every
// time if only one is configured.
type fakeLLM struct {
@@ -56,8 +69,12 @@ func (f *fakeLLM) Chat(context.Context, curator.ChatRequest) (curator.ChatResult
return curator.ChatResult{Text: f.responses[i]}, nil
}
// fakeEngine records what the curator asked it to create. domain/memory's
// own fake engine is unexported in its package's test files, so this one is
// minimal on purpose: the read half exists only to satisfy MemoryEngine.
type fakeEngine struct {
writes []domainmemory.Observation
created []domainmemory.NewRecord
createErr error
}
func (f *fakeEngine) Name() string { return "builtin" }
@@ -70,17 +87,45 @@ func (f *fakeEngine) Health(context.Context) domainmemory.Health {
}
func (f *fakeEngine) Stop(context.Context) error { return nil }
func (f *fakeEngine) Write(_ context.Context, obs domainmemory.Observation) (domainmemory.Record, error) {
f.writes = append(f.writes, obs)
return domainmemory.Record{ID: "id", Identity: obs.Identity, Content: obs.Content}, nil
func (f *fakeEngine) Create(_ context.Context, in domainmemory.NewRecord) (domainmemory.Record, error) {
if f.createErr != nil {
return domainmemory.Record{}, f.createErr
}
f.created = append(f.created, in)
return domainmemory.Record{
ID: domainmemory.RecordID(fmt.Sprintf("r%d", len(f.created))),
Scope: in.Scope,
Kind: in.Kind,
Content: in.Content,
Author: in.Author,
OriginUser: in.OriginUser,
Source: in.Source,
}, nil
}
func (f *fakeEngine) Get(context.Context, domainmemory.Scope, domainmemory.RecordID) (domainmemory.Record, error) {
return domainmemory.Record{}, domainmemory.ErrNotFound
}
func (f *fakeEngine) Query(context.Context, domainmemory.Query) ([]domainmemory.Record, error) {
return nil, nil
}
func (f *fakeEngine) List(context.Context, string) ([]domainmemory.Record, error) { return nil, nil }
func (f *fakeEngine) Forget(context.Context, string) error { return nil }
func (f *fakeEngine) List(context.Context, domainmemory.Scope) ([]domainmemory.Record, error) {
return nil, nil
}
func (f *fakeEngine) Update(context.Context, domainmemory.RecordUpdate) (domainmemory.Record, error) {
return domainmemory.Record{}, domainmemory.ErrNotFound
}
func (f *fakeEngine) Forget(context.Context, domainmemory.Scope, domainmemory.RecordID) error {
return nil
}
func (f *fakeEngine) Revisions(context.Context, domainmemory.Scope, domainmemory.RecordID) ([]domainmemory.Revision, error) {
return nil, nil
}
type fakeEngineSource struct{ engine *fakeEngine }
@@ -140,12 +185,7 @@ func TestCheckReportsTrueWithARecentSession(t *testing.T) {
func TestPassExtractsAndWritesObservations(t *testing.T) {
t.Parallel()
conv := &fakeConversations{
sessions: map[string]time.Time{"s1": time.Unix(1000, 0)},
messages: map[string][]curator.ConversationMessage{
"s1": {{Role: "user", Content: "I prefer tabs over spaces"}},
},
}
conv := newFakeConversations(curator.ConversationMessage{Role: "user", SenderID: "u-42", Content: "I prefer tabs over spaces"})
llm := &fakeLLM{responses: []string{`["prefers tabs over spaces"]`}}
engine := &fakeEngine{}
c := newTestCurator(t, conv, llm, engine)
@@ -160,24 +200,192 @@ func TestPassExtractsAndWritesObservations(t *testing.T) {
if want := time.Unix(1000, 0); !res.Actions[0].At.Equal(want) {
t.Errorf("Actions[0].At = %v, want %v (from the curator's clock, not zero)", res.Actions[0].At, want)
}
if len(engine.writes) != 1 {
t.Fatalf("writes = %#v, want exactly 1", engine.writes)
if len(engine.created) != 1 {
t.Fatalf("created = %#v, want exactly 1", engine.created)
}
got := engine.writes[0]
if got.Identity != "s1" {
t.Errorf("Identity = %q, want the session id, not a user/cross-session identity", got.Identity)
got := engine.created[0]
// The session id is not a scope: what the curator extracted is a fact
// about one person as seen by one agent, and agent-user is the only
// scope that says so.
if want := (domainmemory.Scope{Kind: domainmemory.ScopeAgentUser, Agent: "winter", User: "u-42"}); got.Scope != want {
t.Errorf("Scope = %v, want %v", got.Scope, want)
}
if got.Content != "prefers tabs over spaces" {
t.Errorf("Content = %q, want the extracted fact verbatim", got.Content)
}
}
// TestPassWritesExtractedFactsToAgentUserScope pins the address and
// provenance of every curator write: a later chat turn reads back
// agent-user memory for its own agent and user, and only a record written
// there is visible to it.
func TestPassWritesExtractedFactsToAgentUserScope(t *testing.T) {
t.Parallel()
tests := []struct {
name string
messages []curator.ConversationMessage
response string
wantContents []string
}{
{
name: "one user sender",
messages: []curator.ConversationMessage{{Role: "user", SenderID: "u-42", Content: "I prefer tabs"}},
response: `["prefers tabs over spaces"]`,
wantContents: []string{"prefers tabs over spaces"},
},
{
name: "the same sender across several messages is still one participant",
messages: []curator.ConversationMessage{
{Role: "user", SenderID: "u-42", Content: "I prefer tabs"},
{Role: "assistant", Content: "noted"},
{Role: "user", SenderID: "u-42", Content: "and vim bindings"},
},
response: `["prefers tabs over spaces"]`,
wantContents: []string{"prefers tabs over spaces"},
},
{
name: "an assistant sender beside one identified user sender is still one participant",
messages: []curator.ConversationMessage{
{Role: "user", SenderID: "u-42", Content: "I prefer tabs"},
{Role: "assistant", SenderID: "winter", Content: "noted"},
},
response: `["prefers tabs over spaces"]`,
wantContents: []string{"prefers tabs over spaces"},
},
{
name: "one write per extracted fact",
messages: []curator.ConversationMessage{{Role: "user", SenderID: "u-42", Content: "lots of facts"}},
response: `["fact a","fact b","fact c"]`,
wantContents: []string{"fact a", "fact b", "fact c"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
engine := &fakeEngine{}
c := newTestCurator(t, newFakeConversations(tt.messages...), &fakeLLM{responses: []string{tt.response}}, engine)
res, err := c.Pass(context.Background(), curator.PassInput{})
if err != nil {
t.Fatalf("Pass() = %v, want nil", err)
}
wantScope := domainmemory.Scope{Kind: domainmemory.ScopeAgentUser, Agent: "winter", User: "u-42"}
if len(engine.created) != len(tt.wantContents) {
t.Fatalf("created = %#v, want %d record(s)", engine.created, len(tt.wantContents))
}
for i, want := range tt.wantContents {
got := engine.created[i]
if got.Scope != wantScope {
t.Errorf("created[%d].Scope = %v, want %v", i, got.Scope, wantScope)
}
if got.Kind != "note" {
t.Errorf("created[%d].Kind = %q, want %q", i, got.Kind, "note")
}
if got.Author != Name {
t.Errorf("created[%d].Author = %q, want the producer's own name %q", i, got.Author, Name)
}
if got.OriginUser != domainmemory.IdentityID("u-42") {
t.Errorf("created[%d].OriginUser = %q, want the participant %q", i, got.OriginUser, "u-42")
}
if got.Source != "s1" {
t.Errorf("created[%d].Source = %q, want the session id %q", i, got.Source, "s1")
}
if got.Content != want {
t.Errorf("created[%d].Content = %q, want %q", i, got.Content, want)
}
}
if len(res.Actions) != 1 || res.Actions[0].Type != ActionExtracted {
t.Errorf("Actions = %#v, want one %s", res.Actions, ActionExtracted)
}
})
}
}
// TestPassSkipsSessionsWithoutASingleParticipant pins the fail-closed rule.
// Every case's model response offers a fact, so a write here could only come
// from a guessed participant.
func TestPassSkipsSessionsWithoutASingleParticipant(t *testing.T) {
t.Parallel()
tests := []struct {
name string
messages []curator.ConversationMessage
wantReason string
}{
{
name: "zero senders: a dashboard session carries no per-person identity",
messages: []curator.ConversationMessage{{Role: "user", Content: "hello"}},
wantReason: "no single participant: 0 distinct sender(s), 1 unidentified user message(s)",
},
{
name: "an identified sender beside an unidentified user message: the excerpt's provenance is unknown",
messages: []curator.ConversationMessage{
{Role: "user", SenderID: "u-42", Content: "I prefer tabs"},
{Role: "user", Content: "a message whose channel carries no sender id"},
},
wantReason: "no single participant: 1 distinct sender(s), 1 unidentified user message(s)",
},
{
name: "several distinct senders: a group chat",
messages: []curator.ConversationMessage{
{Role: "user", SenderID: "u-1", Content: "hi"},
{Role: "user", SenderID: "u-2", Content: "hello"},
},
wantReason: "no single participant: 2 distinct sender(s)",
},
{
name: "a sender present only on assistant messages does not count",
messages: []curator.ConversationMessage{{Role: "assistant", SenderID: "winter", Content: "hi"}},
wantReason: "no single participant: 0 distinct sender(s)",
},
{
name: "empty senders among the user messages do not create a participant",
messages: []curator.ConversationMessage{
{Role: "user", Content: "hello"},
{Role: "user", Content: "still nobody"},
},
wantReason: "no single participant: 0 distinct sender(s), 2 unidentified user message(s)",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
engine := &fakeEngine{}
c := newTestCurator(t, newFakeConversations(tt.messages...), &fakeLLM{responses: []string{`["a fact worth keeping"]`}}, engine)
res, err := c.Pass(context.Background(), curator.PassInput{})
if err != nil {
t.Fatalf("Pass() = %v, want nil: an unattributable session is skipped, not a pass failure", err)
}
if len(engine.created) != 0 {
t.Fatalf("created = %#v, want none: the session has no single participant to write for", engine.created)
}
if len(res.Actions) != 1 {
t.Fatalf("Actions = %#v, want exactly one skip", res.Actions)
}
got := res.Actions[0]
if got.Type != ActionSkipped {
t.Errorf("Actions[0].Type = %q, want %q", got.Type, ActionSkipped)
}
if got.Detail != "s1" {
t.Errorf("Actions[0].Detail = %q, want the session id %q", got.Detail, "s1")
}
if got.Reason != tt.wantReason {
t.Errorf("Actions[0].Reason = %q, want %q", got.Reason, tt.wantReason)
}
if want := time.Unix(1000, 0); !got.At.Equal(want) {
t.Errorf("Actions[0].At = %v, want %v (from the curator's clock)", got.At, want)
}
})
}
}
func TestPassTreatsUnparseableResponseAsNothingExtracted(t *testing.T) {
t.Parallel()
conv := &fakeConversations{
sessions: map[string]time.Time{"s1": time.Unix(1000, 0)},
messages: map[string][]curator.ConversationMessage{"s1": {{Role: "user", Content: "hello"}}},
}
conv := newFakeConversations(curator.ConversationMessage{Role: "user", SenderID: "u-42", Content: "hello"})
llm := &fakeLLM{responses: []string{"not json at all"}}
engine := &fakeEngine{}
c := newTestCurator(t, conv, llm, engine)
@@ -189,17 +397,14 @@ func TestPassTreatsUnparseableResponseAsNothingExtracted(t *testing.T) {
if len(res.Actions) != 0 {
t.Fatalf("Actions = %#v, want none", res.Actions)
}
if len(engine.writes) != 0 {
t.Fatalf("writes = %#v, want none", engine.writes)
if len(engine.created) != 0 {
t.Fatalf("created = %#v, want none", engine.created)
}
}
func TestPassEmptyArrayExtractsNothing(t *testing.T) {
t.Parallel()
conv := &fakeConversations{
sessions: map[string]time.Time{"s1": time.Unix(1000, 0)},
messages: map[string][]curator.ConversationMessage{"s1": {{Role: "user", Content: "hi"}}},
}
conv := newFakeConversations(curator.ConversationMessage{Role: "user", SenderID: "u-42", Content: "hi"})
llm := &fakeLLM{responses: []string{`[]`}}
engine := &fakeEngine{}
c := newTestCurator(t, conv, llm, engine)
@@ -213,12 +418,27 @@ func TestPassEmptyArrayExtractsNothing(t *testing.T) {
}
}
func TestPassTreatsASessionWithNoMessagesAsNothingExtracted(t *testing.T) {
t.Parallel()
conv := newFakeConversations()
engine := &fakeEngine{}
c := newTestCurator(t, conv, &fakeLLM{responses: []string{`["a fact worth keeping"]`}}, engine)
res, err := c.Pass(context.Background(), curator.PassInput{})
if err != nil {
t.Fatalf("Pass() = %v, want nil", err)
}
if len(res.Actions) != 0 {
t.Fatalf("Actions = %#v, want none: a session with no messages yields nothing to extract", res.Actions)
}
if len(engine.created) != 0 {
t.Fatalf("created = %#v, want none", engine.created)
}
}
func TestPassCapsObservationsPerSession(t *testing.T) {
t.Parallel()
conv := &fakeConversations{
sessions: map[string]time.Time{"s1": time.Unix(1000, 0)},
messages: map[string][]curator.ConversationMessage{"s1": {{Role: "user", Content: "lots of facts"}}},
}
conv := newFakeConversations(curator.ConversationMessage{Role: "user", SenderID: "u-42", Content: "lots of facts"})
llm := &fakeLLM{responses: []string{`["a","b","c","d","e","f","g"]`}}
engine := &fakeEngine{}
c := newTestCurator(t, conv, llm, engine)
@@ -226,8 +446,15 @@ func TestPassCapsObservationsPerSession(t *testing.T) {
if _, err := c.Pass(context.Background(), curator.PassInput{}); err != nil {
t.Fatalf("Pass() = %v, want nil", err)
}
if len(engine.writes) != maxObservationsPerSession {
t.Fatalf("writes = %d, want capped at %d", len(engine.writes), maxObservationsPerSession)
if len(engine.created) != maxObservationsPerSession {
t.Fatalf("created = %d, want capped at %d", len(engine.created), maxObservationsPerSession)
}
// The cap truncates the tail, and keeps the order the model returned.
want := []string{"a", "b", "c", "d", "e"}
for i, content := range want {
if engine.created[i].Content != content {
t.Errorf("created[%d].Content = %q, want %q", i, engine.created[i].Content, content)
}
}
}
@@ -235,12 +462,16 @@ func TestPassReviewsMultipleSessionsIndependently(t *testing.T) {
t.Parallel()
conv := &fakeConversations{
sessions: map[string]time.Time{"s1": time.Unix(1000, 0), "s2": time.Unix(1000, 0)},
agents: map[string]string{"s1": "winter", "s2": "spring"},
messages: map[string][]curator.ConversationMessage{
"s1": {{Role: "user", Content: "fact for s1"}},
"s2": {{Role: "user", Content: "fact for s2"}},
"s1": {{Role: "user", SenderID: "u-1", Content: "fact for s1"}},
"s2": {{Role: "user", SenderID: "u-2", Content: "fact for s2"}},
},
}
llm := &fakeLLM{responses: []string{`["fact one"]`, `["fact two"]`}}
// One response for both sessions: the map iteration order above is not
// deterministic, so nothing here may depend on which session is asked
// first -- attribution is asserted through Source.
llm := &fakeLLM{responses: []string{`["a fact"]`}}
engine := &fakeEngine{}
c := newTestCurator(t, conv, llm, engine)
@@ -251,15 +482,23 @@ func TestPassReviewsMultipleSessionsIndependently(t *testing.T) {
if len(res.Actions) != 2 {
t.Fatalf("Actions = %#v, want 2 (one per session)", res.Actions)
}
if len(engine.writes) != 2 {
t.Fatalf("writes = %#v, want 2", engine.writes)
if len(engine.created) != 2 {
t.Fatalf("created = %#v, want 2", engine.created)
}
identities := map[string]bool{}
for _, w := range engine.writes {
identities[w.Identity] = true
scopes := map[string]domainmemory.Scope{}
for _, rec := range engine.created {
scopes[rec.Source] = rec.Scope
}
if !identities["s1"] || !identities["s2"] {
t.Errorf("writes scoped to identities %v, want both s1 and s2", identities)
// Each session's own agent addresses its scope, so a session summary
// whose agent id is dropped or swapped writes into another agent's
// memory.
for session, want := range map[string]domainmemory.Scope{
"s1": {Kind: domainmemory.ScopeAgentUser, Agent: "winter", User: "u-1"},
"s2": {Kind: domainmemory.ScopeAgentUser, Agent: "spring", User: "u-2"},
} {
if got := scopes[session]; got != want {
t.Errorf("scope of the write sourced from %s = %v, want %v", session, got, want)
}
}
}
@@ -340,17 +579,17 @@ func TestRegistersAndRunsThroughTheRegistry(t *testing.T) {
}
}
var errLLM = errors.New("model unavailable")
var (
errLLM = errors.New("model unavailable")
errCreate = errors.New("engine refused the write")
)
func TestPassPropagatesALLMFailure(t *testing.T) {
// Unlike an unparseable response, the model call itself failing (the
// provider is down) is a real Pass error, not a "nothing to extract"
// outcome.
t.Parallel()
conv := &fakeConversations{
sessions: map[string]time.Time{"s1": time.Unix(1000, 0)},
messages: map[string][]curator.ConversationMessage{"s1": {{Role: "user", Content: "hi"}}},
}
conv := newFakeConversations(curator.ConversationMessage{Role: "user", SenderID: "u-42", Content: "hi"})
llm := &fakeLLM{err: errLLM}
c := newTestCurator(t, conv, llm, &fakeEngine{})
@@ -358,3 +597,17 @@ func TestPassPropagatesALLMFailure(t *testing.T) {
t.Fatalf("Pass() error = %v, want it to wrap the LLM failure", err)
}
}
func TestPassPropagatesACreateFailure(t *testing.T) {
// The engine refusing a write is a real failure, not "nothing to
// extract": the pass must not report success for a fact it did not
// persist.
t.Parallel()
conv := newFakeConversations(curator.ConversationMessage{Role: "user", SenderID: "u-42", Content: "hi"})
engine := &fakeEngine{createErr: errCreate}
c := newTestCurator(t, conv, &fakeLLM{responses: []string{`["a fact"]`}}, engine)
if _, err := c.Pass(context.Background(), curator.PassInput{}); !errors.Is(err, errCreate) {
t.Fatalf("Pass() error = %v, want it to wrap the engine's create failure", err)
}
}
+26 -117
View File
@@ -1,123 +1,32 @@
package memory
import "regexp"
import domainmemory "github.com/samcharles93/archie-core/internal/domain/memory"
// ThreatLevel indicates the severity of a threat detected by a Scanner.
type ThreatLevel int
// The content scanner moved into the memory engine family
// (internal/domain/memory/scanner.go, docs/prds/memory-engine-unification.md
// §7), where the engine that persists memory applies it. This package's
// legacy Manager still needs the same types until it is deleted in slice 5 of
// that PRD, so they are re-exported here rather than duplicated: the
// behaviour, the patterns and the tests stay exactly where the decision put
// them.
//
// Delete this file with internal/memory.
type (
// ThreatLevel is domainmemory.ThreatLevel.
ThreatLevel = domainmemory.ThreatLevel
// ThreatResult is domainmemory.ThreatResult.
ThreatResult = domainmemory.ThreatResult
// Scanner is domainmemory.Scanner.
Scanner = domainmemory.Scanner
// DefaultScanner is domainmemory.DefaultScanner.
DefaultScanner = domainmemory.DefaultScanner
)
const (
// ThreatNone means no suspicious patterns were found.
ThreatNone ThreatLevel = iota
// ThreatWarn means a potentially problematic pattern was found, but the
// write should be allowed with a warning logged.
ThreatWarn
// ThreatBlock means a dangerous pattern was found and the write should
// be rejected.
ThreatBlock
// ThreatNone is domainmemory.ThreatNone.
ThreatNone = domainmemory.ThreatNone
// ThreatWarn is domainmemory.ThreatWarn.
ThreatWarn = domainmemory.ThreatWarn
// ThreatBlock is domainmemory.ThreatBlock.
ThreatBlock = domainmemory.ThreatBlock
)
// ThreatResult describes a single threat detected during content scanning.
type ThreatResult struct {
// Level indicates the severity: none, warn, or block.
Level ThreatLevel
// Pattern is the regexp that matched (for audit logging).
Pattern string
// Message is a human-readable description of the threat.
Message string
}
// Scanner inspects memory content for dangerous or sensitive patterns before
// the content is persisted. Implementations are pluggable; the zero-value
// DefaultScanner is a reasonable default for production use.
type Scanner interface {
// ScanContent examines content and returns a ThreatResult. A result
// with Level == ThreatNone means the content is clean.
ScanContent(content string) ThreatResult
}
// DefaultScanner is the built-in Scanner implementation. It checks for
// prompt-injection patterns (block-level) and sensitive-data leakage
// (warn-level). The zero value is usable.
type DefaultScanner struct{}
// Compile-time check: DefaultScanner implements Scanner.
var _ Scanner = (*DefaultScanner)(nil)
// ── Patterns ────────────────────────────────────────────────────────────
//
// These patterns are compiled once at init time. They are intentionally
// broad to catch common variants without being so broad that they
// produce false positives on normal memory content.
var (
promptInjectionPatterns = []*regexp.Regexp{
// Override / ignore previous instructions
regexp.MustCompile(`(?i)(ignore|forget|override|disregard)\s+(all\s+)?(the\s+)?(previous|above|prior|earlier)\s+(instructions?|directions?|commands?|prompts?)`),
// Role-change / identity attacks
regexp.MustCompile(`(?i)(you\s+(are|now)\s+|act\s+as\s+|pretend\s+(to\s+be|you\s+are)\s+|roleplay\s+as\s+)(now\s+)?(an?\s+)?(different|new|another|unrestricted|uncensored)`),
// Delimiter injection (common in adversarial prompts)
regexp.MustCompile(`(?i)<\|im_start\|>`),
regexp.MustCompile(`(?i)<\|im_end\|>`),
// System prompt extraction attempts
regexp.MustCompile(`(?i)(print|output|repeat|display|show|tell\s+me)\s+(your\s+)?(system\s+)?(prompt|instructions?|directions?)\s*(exactly|verbatim)?`),
}
sensitiveDataPatterns = []*regexp.Regexp{
// API keys and secrets with common key=value patterns
regexp.MustCompile(`(?i)(api[_\s-]?key|apikey|api[_\s-]?secret)\s*[:=]\s*['"\x60]?[\w-]{8,}['"\x60]?`),
// Passwords in key=value form
regexp.MustCompile(`(?i)(password|passwd|pwd)\s*[:=]\s*['"\x60]?\S{4,}['"\x60]?`),
// Generic secrets and tokens
regexp.MustCompile(`(?i)(secret|token|credential|authorization)\s*[:=]\s*['"\x60]?[\w-]{8,}['"\x60]?`),
// Private key blocks (PEM format)
regexp.MustCompile(`(?i)-----BEGIN\s+(RSA\s+|EC\s+|DSA\s+|OPENSSH\s+)?PRIVATE\s+KEY-----`),
// JWT tokens (base64-encoded header.payload.signature pattern)
regexp.MustCompile(`eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}`),
}
)
// ScanContent examines content against the prompt-injection and
// sensitive-data patterns. Prompt-injection matches produce ThreatBlock;
// sensitive-data matches produce ThreatWarn. The first match wins.
func (s *DefaultScanner) ScanContent(content string) ThreatResult {
for _, p := range promptInjectionPatterns {
if m := p.FindString(content); m != "" {
return ThreatResult{
Level: ThreatBlock,
Pattern: p.String(),
Message: "content matches prompt injection pattern: " + truncateMatch(m),
}
}
}
for _, p := range sensitiveDataPatterns {
if m := p.FindString(content); m != "" {
return ThreatResult{
Level: ThreatWarn,
Pattern: p.String(),
Message: "content may contain sensitive data: " + truncateMatch(m),
}
}
}
return ThreatResult{Level: ThreatNone}
}
// truncateMatch limits the matched string for log/error messages so that
// a long match (e.g. an entire private key) is not echoed in full.
func truncateMatch(s string) string {
const maxMatchLen = 40
if len(s) <= maxMatchLen {
return s
}
return s[:maxMatchLen] + "..."
}