feat(memory): four-scope contract and builtin engine CRUD (unification slice 2) (#886)

* feat(memory): four-scope contract with CRUD and revisions

Seam for memory unification slice 2 (docs/prds/memory-engine-unification.md
§1, §2, §7): internal/domain/memory is now a scope-addressed CRUD contract.

- Scope{Kind,Agent,User} with the four ratified kinds, Validate rejecting a
  component present where it is not required, and a length-prefixed Key so
  opaque channel-native ids cannot collide.
- Subject{AgentID,UserID} with Scopes()/WritableScopes(): global is never
  writable, and an absent user id removes every scope requiring one.
- Store becomes Create/Get/Query/List/Update/Forget/Revisions over
  NewRecord/RecordUpdate/Record/Revision, with ErrNotFound/ErrStaleRevision.
  Observation, Identity and Metadata are deleted.
- The content scanner moves into the family (internal/domain/memory) so every
  engine applies it in Create and Update; internal/memory re-exports the types
  until slice 5 deletes it.
- The markdown store moves to internal/infrastructure/memory/builtin, its
  correct home per organisation.md.

WIP: internal/infrastructure/memory and internal/infrastructure/sessioncurator
are converted in the lanes that follow this commit.

* feat(memory): builtin engine CRUD, retained revisions and the curator's agent-user write

Slice 2 implementation (docs/prds/memory-engine-unification.md §1, §2, §6,
§7), on top of the contract seam in the previous commit.

- internal/infrastructure/memory: per-scope markdown heads with a JSON
  provenance marker, per-scope HISTORY.md retaining superseded and deleted
  states, Update superseding before replacing the live block, ErrStaleRevision
  via RecordUpdate.Expected, idempotent Forget, and the family scanner applied
  in Create and Update. Scope directories are hex(sha256(Scope.Key())).
- internal/infrastructure/sessioncurator: the participant is derived from the
  session's user-role senders; exactly one writes agent-user scope, zero or
  several write nothing and return ActionSkipped naming the count.
- docs/architecture: the settled decision recorded in migration-decisions §5
  (superseding the 2026-09-02 marker entry) and agent-system.md pointing at it.

Draft: to be squashed and reviewed before publication.

* test(curator): conform the registry's fake memory engine to the CRUD contract

The fake still implemented the pre-CRUD Store surface (Write taking an
Observation, List/Forget keyed by a bare string), so the package's test
binary no longer compiles: fakeMemoryEngine is missing Create, Get, Update
and Revisions, and domainmemory.Observation no longer exists. That took
`go fix ./...`, `go vet ./...` and `go test ./...` down with it, which is
what `task check` runs first.

Missed by the same commit that updated internal/infrastructure/sessioncurator's
own fake to the new interface. Repaired on the fake, not by loosening the
interface: the fake is the only thing out of date.

* fix(sessioncurator): refuse to attribute a session with unidentified user messages

sessionParticipant skipped user-role messages with an empty SenderID and
counted only the identified ones, so a session holding one identified sender
plus any number of unidentified user messages yielded that sender as the
single participant. Those messages are still part of the excerpt the
extraction model reads, so facts derived from content of unknown provenance
were written into the identified sender's agent-user scope: user A's memory
can end up crediting content that may belong to someone else. It is exactly
the case the function's own doc comment claims to handle, which it only
caught when no message at all was identified.

An unidentified user message now makes the session unattributable outright,
beside any number of identified senders, and the skip action's reason names
the count instead of reporting a sender count that no longer explains the
skip. Assistant-role messages are unaffected: they carry the bot, so they
still cannot manufacture a participant.

Fixed in sessionParticipant rather than at the write, because it is the
single producer of the attribution decision: it now returns the participant
or the reason there is none, so the caller cannot print a reason that
disagrees with what it decided. The fail-closed path (nothing written, one
ActionSkipped) and the extraction flow are unchanged.

* fix(memory): carry record content through the block format intact

builtin.Store's format ends a block at a blank line and starts a section at a
"## " line, but storeContent only trimmed the content and rejected an empty
one. Content holding either delimiter was accepted, written, and answered back
to the caller in full -- then read back truncated at the delimiter by Get,
List, or a process restart, with the remainder unaddressable and no error
anywhere along the path.

Fixed at the producer, in the render/parse pair that owns the block format:
renderBlock escapes the two delimiter shapes (an empty line becomes a single
backslash, a line already starting with a backslash or "## " gains one) and
parseBlocks undoes it on the way back, so a record's content survives the
write whole. Rejecting the content instead would have made an ordinary
multi-paragraph note or a quoted heading unrepresentable; escaping every line
would have changed the on-disk shape of every record. Only the lines that need
it are escaped, so a one-paragraph record is byte-identical to what the
unfixed engine wrote and the document stays hand-editable.

Reachable from both producers: the chat tool's note and the curator's
model-extracted fact are free text, and neither is constrained to one
paragraph.

Tests: TestBuiltinEngineRenderedBlockRoundTripsContent asserts
parseBlocks(renderBlock(c)) == c for eleven content shapes, and
TestBuiltinEngineContentSurvivesTheWriteAndAReopen carries the same shapes
through Create, Get, and a second engine reading the file a restart would.

* fix(memory): compare Update's expected revision against the file, not the cache

Update read the record to compare against out of findBlock(stores.live, ...)
-- the engine's cached copy of the scope document -- and checked Expected
against that copy. This engine caches one document per scope and rewrites it
whole, so a second engine over the same scope file (archied and the gateway,
or a restart) holds a copy taken when it last read the file: its view of the
record's revision is superseded, Expected matches the revision it remembers,
ErrStaleRevision never fires, and the rewrite that follows writes that stale
copy back over the file -- discarding the other writer's update and every
record written since the copy was loaded. The field existed and could not
fire.

Reload the scope document before anything is decided from it, so the check
answers about the revision actually on disk and the rewrite starts from the
current document. That is the smaller of the two fixes the PRD leaves open
("Not determined": a reload-before-mutate or an advisory lock); no locking
scheme, no retry loop, and no change to the read paths.

Still unsolved, and deliberately: the whole-document rewrite is
last-writer-wins between the reload and the write, so two engines writing one
scope concurrently can still lose one of the two writes, and HISTORY.md is
rewritten from the cached copy the same way.

Tests: TestBuiltinEngineSecondEngineCannotReuseAStaleRevision asserts the
second engine gets ErrStaleRevision and the first writer's state survives;
TestBuiltinEngineUpdateDoesNotEraseRecordsItHasNotLoaded asserts the record
written after the second engine loaded is still there after its update.

* fix(memory): emit the scanner's warning instead of dropping it

scanContent returned nil for every result that was not ThreatBlock, so a
ThreatWarn result -- documented in the scanner as "allow the write but emit a
warning", and what every sensitive-data match produces (API keys, passwords,
tokens, private keys) -- was stored with no audit signal anywhere. The
comment justified the silence by pointing at Bind, but Bind's argument is
about Registrar.Events: internal/events declares no memory kind and the
composition binds no memory event sink, so an invented kind would look wired
and not be. A scanner warning is a diagnostic rather than an event, and there
was nothing else to carry it: the registrar had no logger.

Bind now takes Registrar.Log, optional and mirroring the curator family's
Registrar for the same reason, scanContent emits the warn-level result
through it before the write is allowed, and bootstrap passes the daemon's
logger so the line reaches archied's configured sinks instead of only the
process default. The write is still allowed: only the silence changed.

Tests: TestBuiltinEngineScanWarnsWithoutRefusingTheWrite runs Create and
Update over warn, clean and block content, asserting the write lands either
way, that a warn-level scan result logs exactly one warning carrying the
scanner's own message, that clean content logs none, and that a refused
block-level match logs none.

* test(memory): carry every content shape through Update and Revisions

Update replaces the block it read out of the document by that text, and
retains the superseded state in HISTORY.md, so an escaped block has to be
found, superseded and retained exactly like an unescaped one. Covered here
rather than left to the Create path: without the escaping, every shape in
this table came back truncated after an update, including the plain
one-paragraph record whose replacement holds a blank line.

The test asserts values only: the updated record as a second engine reads it
off disk, and the superseded content as Revisions returns it.

* style(memory): apply the linter's gofumpt form to the Update revision error

golangci-lint's bundled gofumpt wants the closing paren of that wrapped
fmt.Errorf on its own line; the standalone gofumpt binary the fmt task uses
(0.7.0, a different version) leaves it as it was. The text is unchanged from
before the engine fixes in this branch, so this is pre-existing drift inside a
function these commits already touch -- taken so the file is clean under both
formatters rather than only one.
This commit is contained in:
sam authored and GitHub committed 2026-09-19 11:13:21 +10:00
1 parent cd9399f705
commit 7ee5025d3c
23 files changed
+3578 -607

No files matched your search

+5 -3
View File
@@ -95,9 +95,11 @@ because both users interact with the same Agent.
Shared memory does not erase provenance. Every memory record retains its scope, Shared memory does not erase provenance. Every memory record retains its scope,
author or acting identity, originating user when applicable, source, and author or acting identity, originating user when applicable, source, and
revision history. The exact rules for selecting a write scope, granting access revision history. The rules for selecting a write scope, granting access
to global and user-wide memory, and authorizing explicit cross-scope sharing to global and user-wide memory, and authorizing explicit cross-scope sharing are
require a focused memory design decision. settled by [`docs/prds/memory-engine-unification.md`](../prds/memory-engine-unification.md)
and recorded in [`migration-decisions.md`](migration-decisions.md#5-memory-placement-and-storage)
section 5.
An Agent interaction may answer conversationally, use a tool or capability, read An Agent interaction may answer conversationally, use a tool or capability, read
or send messages, retain information, or propose durable workflow-backed work. or send messages, retain information, or propose durable workflow-backed work.
+64 -7
View File
@@ -242,14 +242,18 @@ The four required scopes are fixed:
- Agent-user relationship. - Agent-user relationship.
A focused review of `internal/memory`, its providers, consumers, persistence, A focused review of `internal/memory`, its providers, consumers, persistence,
and tests must decide: and tests had these decisions:
- `internal/domain/memory` versus ownership within the Agent domain; - `internal/domain/memory` versus ownership within the Agent domain — settled
- the authoritative memory record and revision model; (2026-09-02, below);
- provider and infrastructure boundaries; - the authoritative memory record and revision model — settled by the PRD
- retrieval and access enforcement; (2026-09-16, below);
- migration of existing memory data; - provider and infrastructure boundaries — settled by the PRD (2026-09-16,
- provenance representation. below);
- retrieval and access enforcement — settled by the PRD (2026-09-16, below);
ranked retrieval remains open (below);
- migration of existing memory data — open: existing data is not migrated;
- provenance representation — settled by the PRD (2026-09-16, below).
Conversation branches do not create another memory scope. A memory action has Conversation branches do not create another memory scope. A memory action has
the same scoped effect regardless of which Conversation originated it. the same scoped effect regardless of which Conversation originated it.
@@ -295,6 +299,59 @@ memory model in general or a future non-file engine's model:
invalid `Kind` fails `Write` with the same validation error `memory_edit` invalid `Kind` fails `Write` with the same validation error `memory_edit`
already returns for an invalid section today, not a new error shape. already returns for an invalid section today, not a new error shape.
**Memory engine: scopes, CRUD with retained revisions, and store placement —
DECIDED (2026-09-16).** Supersedes the 2026-09-02 "Authoritative record and
revision model for the builtin engine" entry above, whose `<!--mem:<ulid>-->`
live marker cannot carry provenance. Authority:
[`docs/prds/memory-engine-unification.md`](../prds/memory-engine-unification.md)
(Approved); the contract is `internal/domain/memory/contract.go`. This register
supersedes; it does not erase.
- **Four typed scopes, caller-named.** `internal/domain/memory` is addressed by
`Scope{Kind, Agent, User}` over `global`, `agent`, `user` and `agent-user`,
with `Validate` rejecting a component present where it is not required. A
caller names the read and write sets (`Subject.Scopes()` /
`WritableScopes()`; global is never model-writable) and the engine applies no
policy of its own: **access control is naming a scope, and nothing else.** An
engine that enforced policy would need to know about channels, sessions and
bindings, and every backend would reimplement it. `Scope.Key()` length-prefixes
each component, so two scopes can never share a key even when an opaque
channel-native id embeds the separator.
- **The caller resolves identity; the engine never does.** Resolution happens in
the gateway turn path, where the native sender id and the session are both in
hand, and is supplied per channel by the composition. A webhook's sender id is
a route path, not a person, and treating it as one would give a URL an
identity. An absent user id yields global and agent scopes only: "show none of
it", never a wider fallback.
- **CRUD supersedes, never overwrites.** `Store` is `Create`/`Get`/`Query`/
`List`/`Update`/`Forget`/`Revisions` over the four scopes. `Update` appends the
superseded state to the scope's `HISTORY.md` before replacing the live block;
`Update.Expected` names the revision being replaced and returns
`ErrStaleRevision` on mismatch, making a lost update between two processes
sharing one scope file detectable. `Forget` records a deleted revision before
removing the live block, so a record's provenance outlives its content; `Get`
and `Forget` take a scope, because an id alone does not locate one.
- **Provenance representation.** The live marker becomes JSON, since a
`<!--mem:<ulid>-->` line cannot carry provenance, and `Record` retains the
provenance `agent-system.md` requires — scope, author, originating user,
source, and revision history. `Metadata` is dropped: it never round-tripped.
- **Store and scanner placement.** The markdown store relocates to
`internal/infrastructure/memory/builtin`, its correct home under
`organisation.md`'s "a package owns its on-disk format end-to-end". The content
scanner moves into `internal/domain/memory` and is applied in `Create` and
`Update` — the single choke point every producer crosses, so curator-extracted
content is covered too.
**Open in this section.** Ranked retrieval is deferred: `Query.Text` stays in
the contract, unused by the prompt path. Existing memory data is not migrated:
`<workDir>/memory` and `<workDir>/memory-engine` are left on disk, unreferenced.
The PRD leaves these undetermined and they remain open here: group-chat
extraction, where the curator fails closed on mixed-sender sessions; two
processes (`archied` and `archie-gateway`) holding one scope file, where
`Update.Expected` detects but does not prevent a lost block; unbounded
`HISTORY.md` with no compaction policy; whether the per-scope size bound is
right; and whether the dashboard should carry a user identity.
### 6. Runtime and process boundaries — DECIDED ### 6. Runtime and process boundaries — DECIDED
`archied` is the native resident orchestration and interactive-chat process. `archied` is the native resident orchestration and interactive-chat process.
+3 -1
View File
@@ -1112,7 +1112,9 @@ func (b *boot) setupMemory() error {
// same files while both exist. // same files while both exist.
func (b *boot) setupMemoryEngine() error { func (b *boot) setupMemoryEngine() error {
cfg, log := b.cfg, b.log cfg, log := b.cfg, b.log
registry := domainmemory.NewRegistry(domainmemory.Registrar{}) // The logger is what carries an engine's scanner warning: warn allows
// the write, so the log line is the whole audit trail for it.
registry := domainmemory.NewRegistry(domainmemory.Registrar{Log: log})
switch cfg.Memory.Engine { switch cfg.Memory.Engine {
case infraMemory.EngineName, "": case infraMemory.EngineName, "":
+1 -1
View File
@@ -39,8 +39,8 @@ import (
"github.com/samcharles93/archie-core/internal/gateway" "github.com/samcharles93/archie-core/internal/gateway"
"github.com/samcharles93/archie-core/internal/infrastructure/configuration" "github.com/samcharles93/archie-core/internal/infrastructure/configuration"
"github.com/samcharles93/archie-core/internal/infrastructure/configuration/overlay" "github.com/samcharles93/archie-core/internal/infrastructure/configuration/overlay"
"github.com/samcharles93/archie-core/internal/infrastructure/memory/builtin"
"github.com/samcharles93/archie-core/internal/logging" "github.com/samcharles93/archie-core/internal/logging"
"github.com/samcharles93/archie-core/internal/memory/builtin"
"github.com/samcharles93/archie-core/internal/plugin" "github.com/samcharles93/archie-core/internal/plugin"
"github.com/samcharles93/archie-core/internal/secret" "github.com/samcharles93/archie-core/internal/secret"
"github.com/samcharles93/archie-core/internal/storage" "github.com/samcharles93/archie-core/internal/storage"
+19 -3
View File
@@ -61,18 +61,34 @@ func (fakeMemoryEngine) Health(context.Context) domainmemory.Health {
return domainmemory.Health{Status: domainmemory.HealthHealthy} return domainmemory.Health{Status: domainmemory.HealthHealthy}
} }
func (fakeMemoryEngine) Stop(context.Context) error { return nil } func (fakeMemoryEngine) Stop(context.Context) error { return nil }
func (fakeMemoryEngine) Write(context.Context, domainmemory.Observation) (domainmemory.Record, error) {
func (fakeMemoryEngine) Create(context.Context, domainmemory.NewRecord) (domainmemory.Record, error) {
return domainmemory.Record{}, nil return domainmemory.Record{}, nil
} }
func (fakeMemoryEngine) Get(context.Context, domainmemory.Scope, domainmemory.RecordID) (domainmemory.Record, error) {
return domainmemory.Record{}, domainmemory.ErrNotFound
}
func (fakeMemoryEngine) Query(context.Context, domainmemory.Query) ([]domainmemory.Record, error) { func (fakeMemoryEngine) Query(context.Context, domainmemory.Query) ([]domainmemory.Record, error) {
return nil, nil return nil, nil
} }
func (fakeMemoryEngine) List(context.Context, string) ([]domainmemory.Record, error) { func (fakeMemoryEngine) List(context.Context, domainmemory.Scope) ([]domainmemory.Record, error) {
return nil, nil
}
func (fakeMemoryEngine) Update(context.Context, domainmemory.RecordUpdate) (domainmemory.Record, error) {
return domainmemory.Record{}, domainmemory.ErrNotFound
}
func (fakeMemoryEngine) Forget(context.Context, domainmemory.Scope, domainmemory.RecordID) error {
return nil
}
func (fakeMemoryEngine) Revisions(context.Context, domainmemory.Scope, domainmemory.RecordID) ([]domainmemory.Revision, error) {
return nil, nil return nil, nil
} }
func (fakeMemoryEngine) Forget(context.Context, string) error { return nil }
// testConversations is a fake ConversationSource with one session and no // testConversations is a fake ConversationSource with one session and no
// messages -- enough for the presence/absence tests, which never call // messages -- enough for the presence/absence tests, which never call
+327 -66
View File
@@ -1,24 +1,32 @@
// Package memory is the memory engine family: a pluggable backend for // Package memory is the memory engine family: the authoritative store for
// durable observations about a session or identity, so an external memory // durable observations, addressed by typed scope. See
// service (Honcho, or a future alternative) can be bolted on without // docs/prds/memory-engine-unification.md for the decision this contract
// touching the agent loop. See epic archie-core-1786637499161-356-e424e40d. // implements.
//
// One engine, four scopes. A caller names the scopes it may read or write
// (Scope, Subject) and the engine applies no policy of its own: an engine
// that enforced access would need to know about channels, sessions and
// bindings, and every backend would reimplement it. Isolation then falls out
// mechanically, because ScopeAgent{a} and ScopeAgent{b} are different
// storage keys.
// //
// The family follows the strict plugin engine rule: a typed contract // The family follows the strict plugin engine rule: a typed contract
// (MemoryEngine) with real domain operations (write, query, list, forget — // (MemoryEngine) with real domain operations (create, read, update, delete,
// not just Name/Version), an owning registry (Registry) with // revisions — not just Name/Version), an owning registry (Registry) with
// start/health/stop and shutdown ordering, and narrow typed host access // start/health/stop and shutdown ordering, and narrow typed host access
// (Registrar) — an engine never receives the daemon or an untyped hook map. // (Registrar) — an engine never receives the daemon or an untyped hook map.
// //
// This package defines the contract only. internal/memory's existing // This package owns the contract and the family's policy that is independent
// MemoryProvider and Manager predate the domain migration and are the // of any backend (the content scanner). internal/memory's MemoryProvider and
// current runtime path; they are not extended in place (see // Manager predate the domain migration and are the legacy runtime path,
// archie-core-1786637490069-11-e245c170's notes) and are superseded by this // deleted in slice 5 of the same PRD; nothing here depends on them.
// family as engines land against it.
package memory package memory
import ( import (
"context" "context"
"errors" "errors"
"fmt"
"strconv"
"time" "time"
) )
@@ -59,23 +67,186 @@ type Lifecycle interface {
Stop(ctx context.Context) error Stop(ctx context.Context) error
} }
// Store is the real domain operations every memory backend implements — // ScopeKind names one of the four durable memory scopes
// what the epic requires beyond identity metadata. Split out from // (docs/architecture/agent-system.md, "Memory scopes").
// MemoryEngine so the two concerns (storage vs. identity/lifecycle) stay type ScopeKind string
// separately named and independently testable.
const (
// ScopeGlobal is shared across the application. Never model-writable:
// writing it is an operator action. It is part of the contract because
// the read set includes it.
ScopeGlobal ScopeKind = "global"
// ScopeAgent is private to one Agent. Agent-wide is not a synonym for
// shared; ScopeAgent{a} and ScopeAgent{b} are different keys.
ScopeAgent ScopeKind = "agent"
// ScopeUser is one user, across every Agent serving them.
ScopeUser ScopeKind = "user"
// ScopeAgentUser is one Agent's relationship with one user: the only
// home for relationship facts, and the scope the session curator
// writes.
ScopeAgentUser ScopeKind = "agent-user"
)
// AgentID and IdentityID are opaque. This package never resolves them,
// canonicalises them, or invents one from a channel: resolution happens
// where the channel's native identity is in hand (the gateway turn path),
// because a webhook's sender id is a route path and treating it as a person
// would give a URL an identity.
type (
AgentID string
IdentityID string
)
// Scope addresses one region of memory. Which components are required is a
// function of Kind, and Validate is the only place that rule lives.
type Scope struct {
Kind ScopeKind
Agent AgentID
User IdentityID
}
// Validate rejects an unknown kind, a missing component, and — just as
// importantly — a component present where it is not required, so no two
// spellings address the same region.
func (s Scope) Validate() error {
switch s.Kind {
case ScopeGlobal:
if s.Agent != "" || s.User != "" {
return fmt.Errorf("memory scope global: takes no agent or user, got agent %q user %q", s.Agent, s.User)
}
case ScopeAgent:
if s.Agent == "" {
return errors.New("memory scope agent: agent must not be empty")
}
if s.User != "" {
return fmt.Errorf("memory scope agent: takes no user, got %q", s.User)
}
case ScopeUser:
if s.User == "" {
return errors.New("memory scope user: user must not be empty")
}
if s.Agent != "" {
return fmt.Errorf("memory scope user: takes no agent, got %q", s.Agent)
}
case ScopeAgentUser:
if s.Agent == "" || s.User == "" {
return fmt.Errorf("memory scope agent-user: agent and user both required, got agent %q user %q", s.Agent, s.User)
}
default:
return fmt.Errorf("memory scope: unknown kind %q", s.Kind)
}
return nil
}
// Key returns the scope's canonical storage key. Each component is
// length-prefixed, so two different scopes can never share a key even when
// an id embeds the separator byte — which matters because AgentID and
// IdentityID are opaque and come from channels this package does not
// control.
//
// An invalid scope returns "", which a storage key can never legitimately
// be: callers must Validate first. Every engine path does.
func (s Scope) Key() string {
switch s.Kind {
case ScopeGlobal:
return "global"
case ScopeAgent:
return "agent:" + lengthPrefixed(string(s.Agent))
case ScopeUser:
return "user:" + lengthPrefixed(string(s.User))
case ScopeAgentUser:
return "agent-user:" + lengthPrefixed(string(s.Agent)) + ":" + lengthPrefixed(string(s.User))
default:
return ""
}
}
// String renders the scope for logs and error messages in Key's canonical
// form.
func (s Scope) String() string { return s.Key() }
// lengthPrefixed renders s as "<byteLen>:<s>", so a component containing ':'
// (or any other byte) cannot be confused with the boundary between
// components.
func lengthPrefixed(s string) string {
return strconv.Itoa(len(s)) + ":" + s
}
// Subject is a turn's resolved identity: one value, supplied by the
// composition per channel, never inferred by the engine.
type Subject struct {
// AgentID is the Agent serving the turn. Required in practice; an empty
// AgentID simply yields no agent-addressed scopes.
AgentID AgentID
// UserID is the initiating user's identity. Empty when the channel
// carries no per-person identity (the dashboard, a webhook) — which
// means "show none of it", never "fall back to a wider scope".
UserID IdentityID
}
// Scopes returns the scopes a read for this subject names, narrowest first:
// agent-user, agent, user, then global last, because global is the only
// scope every subject shares.
//
// The read path hands exactly this set to the engine and the engine unions
// nothing beyond it.
func (s Subject) Scopes() []Scope {
return append(s.WritableScopes(), Scope{Kind: ScopeGlobal})
}
// WritableScopes returns the scopes a write for this subject may name —
// Scopes() minus global, which is never model-writable. A component the
// subject does not carry removes every scope that requires it.
func (s Subject) WritableScopes() []Scope {
var scopes []Scope
if s.AgentID != "" && s.UserID != "" {
scopes = append(scopes, Scope{Kind: ScopeAgentUser, Agent: s.AgentID, User: s.UserID})
}
if s.AgentID != "" {
scopes = append(scopes, Scope{Kind: ScopeAgent, Agent: s.AgentID})
}
if s.UserID != "" {
scopes = append(scopes, Scope{Kind: ScopeUser, User: s.UserID})
}
return scopes
}
// Store is the real domain operations every memory backend implements.
// Split out from MemoryEngine so the two concerns (storage vs.
// identity/lifecycle) stay separately named and independently testable.
//
// Every method that touches one record takes the record's Scope: a record id
// alone does not locate a scope, and an engine that had to search for one
// would be reimplementing the caller's addressing.
type Store interface { type Store interface {
// Write records one observation and returns the stored Record, // Create records one new memory and returns the stored Record,
// including the identifier Forget later needs. // including the identifier Get/Update/Forget/Revisions later need.
Write(ctx context.Context, obs Observation) (Record, error) Create(ctx context.Context, in NewRecord) (Record, error)
// Query returns records relevant to q, most relevant or most recent // Get returns one record by id within scope. A missing record is
// first — ordering is engine-defined and not part of this contract. // ErrNotFound.
Get(ctx context.Context, scope Scope, id RecordID) (Record, error)
// Query returns the live records of q.Scopes, grouped in the order
// those scopes were named and ordered within a scope by the engine
// (most recent first is the expected choice). Heads only: superseded
// states are Revisions.
Query(ctx context.Context, q Query) ([]Record, error) Query(ctx context.Context, q Query) ([]Record, error)
// List returns every record for identity, for inspection and // List returns every live record of one scope, for inspection and
// deletion flows rather than relevance-ranked recall. // deletion flows rather than relevance-ranked recall. Heads only.
List(ctx context.Context, identity string) ([]Record, error) List(ctx context.Context, scope Scope) ([]Record, error)
// Forget permanently removes one record. Forgetting an id that does // Update supersedes a record's content and returns the new state. The
// not exist is not an error: the end state (id absent) already holds. // superseded state is retained and readable through Revisions.
Forget(ctx context.Context, id string) error // Unaddressed, or already superseded when RecordUpdate.Expected is
// set, returns ErrNotFound or ErrStaleRevision respectively.
Update(ctx context.Context, in RecordUpdate) (Record, error)
// Forget removes one record's live state, retaining a deleted revision
// so its provenance outlives its content. Forgetting a record that
// does not exist is not an error: the end state (id absent) already
// holds.
Forget(ctx context.Context, scope Scope, id RecordID) error
// Revisions returns one record's retained states, oldest first,
// including the deleted revision Forget records. A record that never
// existed is not an error: the result is empty.
Revisions(ctx context.Context, scope Scope, id RecordID) ([]Revision, error)
} }
// MemoryEngine is the typed contract every memory backend implements. // MemoryEngine is the typed contract every memory backend implements.
@@ -93,64 +264,154 @@ type MemoryEngine interface {
Bind(host Registrar) Bind(host Registrar)
} }
// Observation is one fact to remember, submitted for writing. // RecordID identifies one record within its scope, stable for the record's
type Observation struct { // whole life: assigned once at Create, independent of content, so an edit
// Identity is whose memory this is about (a session id, a user id — // somewhere else can never orphan an id.
// engine-defined, but always required so Query/List/Forget have a type RecordID string
// scope to operate within).
Identity string // NewRecord is one memory submitted for creation. Revision, timestamps and
// Kind is a freeform category ("preference", "fact", "correction"). // id are stamped by the engine; Scope is what the record will be addressed
Kind string // by for the rest of its life.
// Content is the observation itself. type NewRecord struct {
Scope Scope
Kind string
Content string Content string
// At is when the observation was made. Zero means the engine should // Author is the acting identity that wrote the record — an agent id, or
// stamp it with the current time. // a producer's own name ("operator", "session-memory curator"). Free
At time.Time // text: the engine retains it, it does not interpret it.
// Metadata carries engine-agnostic key/value context alongside the Author string
// observation (e.g. source, confidence). // OriginUser is the user whose interaction produced the record, when
Metadata map[string]string // one is known. Distinct from Scope.User: a curator writing
// agent-user scope has both, but a record about a user may be written
// into agent scope by an operator.
OriginUser IdentityID
// Source is the producer's own label for where the record came from
// (a tool name, a curator pass, a session id). Free text.
Source string
} }
// Validate checks the fields every engine needs regardless of backend: // Validate checks what every engine needs before writing: an addressable
// Identity scopes every other operation, and Content is the whole point // scope and content. Kind's format belongs to the backend (the builtin
// of writing. // engine maps it to a section name), so it is validated there.
func (o Observation) Validate() error { //
if o.Identity == "" { // Implementations must also scan Content before persisting it — see
return errors.New("memory observation: identity must not be empty") // ScanContent and docs/prds/memory-engine-unification.md §7.
func (n NewRecord) Validate() error {
if err := n.Scope.Validate(); err != nil {
return err
} }
if o.Content == "" { if n.Content == "" {
return errors.New("memory observation: content must not be empty") return errors.New("memory record: content must not be empty")
} }
return nil return nil
} }
// Query asks an engine to recall observations relevant to Text within // Query asks an engine to recall records within the caller-named scopes.
// Identity's scope.
type Query struct { type Query struct {
Identity string // Scopes is the read set, narrowest first (Subject.Scopes()). The
Text string // engine unions exactly these and applies no policy of its own.
Scopes []Scope
// Text filters by content substring when non-empty. It stays unused by
// the prompt path; ranked retrieval is a deliberately open question.
Text string
// Limit bounds the number of records returned. Zero lets the engine // Limit bounds the number of records returned. Zero lets the engine
// choose a default; there is no way to request "unlimited" — List // choose a default; there is no way to request "unlimited" — List
// exists for that. // exists for that.
Limit int Limit int
} }
// Validate checks the field every engine needs to scope a query: Identity. // Validate rejects an empty or invalid read set, and a negative limit.
// Text may be empty (some engines support a scope-only recall), so it is
// not required here.
func (q Query) Validate() error { func (q Query) Validate() error {
if q.Identity == "" { if len(q.Scopes) == 0 {
return errors.New("memory query: identity must not be empty") return errors.New("memory query: at least one scope is required")
}
for _, scope := range q.Scopes {
if err := scope.Validate(); err != nil {
return fmt.Errorf("memory query: %w", err)
}
}
if q.Limit < 0 {
return fmt.Errorf("memory query: limit must not be negative, got %d", q.Limit)
} }
return nil return nil
} }
// Record is one stored observation as returned by Write, Query, or List. // RecordUpdate supersedes one record's content. It deliberately cannot name
type Record struct { // a Scope: a record may never move between scopes, so the API makes that
ID string // unrepresentable rather than validating it.
Identity string type RecordUpdate struct {
Kind string // Scope and ID address the record being superseded.
Content string Scope Scope
At time.Time ID RecordID
Metadata map[string]string // Content replaces the live content. Required.
Content string
// Expected optionally names the revision this update supersedes. Zero
// means "whatever is live". A mismatch returns ErrStaleRevision, which
// matters because two processes (archied and archie-gateway) can hold
// the same scope's file.
Expected int
// Author and Source are refreshed on the new revision, with the same
// meaning as in NewRecord.
Author string
Source string
} }
// Validate checks the address and the replacement content.
func (u RecordUpdate) Validate() error {
if err := u.Scope.Validate(); err != nil {
return err
}
if u.ID == "" {
return errors.New("memory record update: id must not be empty")
}
if u.Content == "" {
return errors.New("memory record update: content must not be empty")
}
if u.Expected < 0 {
return fmt.Errorf("memory record update: expected revision must not be negative, got %d", u.Expected)
}
return nil
}
// Record is one stored memory as returned by Create, Get, Query, List or
// Update. It carries the provenance agent-system.md requires be retained:
// scope, kind, content, revision, author, originating user, source, and
// timestamps.
type Record struct {
ID RecordID
Scope Scope
Kind string
Content string
Revision int
Author string
OriginUser IdentityID
Source string
CreatedAt time.Time
UpdatedAt time.Time
}
// Revision is one retained state of a record: a state that was live and has
// been superseded, or the deletion marker Forget records. The record's own
// Revision field is the revision this state carried while live.
type Revision struct {
Record Record
// SupersededAt is when this state stopped being live (replaced by an
// Update, or removed by a Forget).
SupersededAt time.Time
// Deleted marks the revision Forget recorded: Record.Content is the
// last live content, retained so provenance outlives it.
Deleted bool
}
// Store errors. Both cross a producer/consumer boundary, and both are
// expected conditions rather than failures: a missing record and a losing
// race between two processes holding one scope.
var (
// ErrNotFound is returned when a record does not exist in the named
// scope. Get and Update return it; Forget and Revisions do not, because
// their end state already holds.
ErrNotFound = errors.New("memory record not found")
// ErrStaleRevision is returned when RecordUpdate.Expected names a
// revision that is no longer live.
ErrStaleRevision = errors.New("memory record revision is stale")
)
+260 -16
View File
@@ -1,39 +1,271 @@
package memory package memory
import "testing" import (
"strings"
"testing"
)
func TestObservationValidateRequiresIdentityAndContent(t *testing.T) { func TestScopeValidate(t *testing.T) {
t.Parallel() t.Parallel()
tests := []struct { tests := []struct {
name string name string
obs Observation scope Scope
wantErr bool wantErr string // "" means valid; otherwise a substring of the error
}{ }{
{"empty identity", Observation{Content: "x"}, true}, {"global", Scope{Kind: ScopeGlobal}, ""},
{"empty content", Observation{Identity: "u1"}, true}, {"agent", Scope{Kind: ScopeAgent, Agent: "a1"}, ""},
{"both empty", Observation{}, true}, {"user", Scope{Kind: ScopeUser, User: "u1"}, ""},
{"valid", Observation{Identity: "u1", Content: "x"}, false}, {"agent-user", Scope{Kind: ScopeAgentUser, Agent: "a1", User: "u1"}, ""},
{"unknown kind", Scope{Kind: "session"}, "unknown kind"},
{"empty kind", Scope{}, "unknown kind"},
{"global rejects agent", Scope{Kind: ScopeGlobal, Agent: "a1"}, "takes no agent or user"},
{"global rejects user", Scope{Kind: ScopeGlobal, User: "u1"}, "takes no agent or user"},
{"agent needs an agent", Scope{Kind: ScopeAgent}, "agent must not be empty"},
{"agent rejects a user", Scope{Kind: ScopeAgent, Agent: "a1", User: "u1"}, "takes no user"},
{"user needs a user", Scope{Kind: ScopeUser}, "user must not be empty"},
{"user rejects an agent", Scope{Kind: ScopeUser, Agent: "a1", User: "u1"}, "takes no agent"},
{"agent-user needs an agent", Scope{Kind: ScopeAgentUser, User: "u1"}, "both required"},
{"agent-user needs a user", Scope{Kind: ScopeAgentUser, Agent: "a1"}, "both required"},
{"agent-user needs both", Scope{Kind: ScopeAgentUser}, "both required"},
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() t.Parallel()
err := tt.obs.Validate() err := tt.scope.Validate()
if (err != nil) != tt.wantErr { switch {
t.Fatalf("Validate() = %v, wantErr %v", err, tt.wantErr) case tt.wantErr == "" && err != nil:
t.Fatalf("Validate() = %v, want nil", err)
case tt.wantErr != "" && err == nil:
t.Fatalf("Validate() = nil, want an error mentioning %q", tt.wantErr)
case tt.wantErr != "" && !strings.Contains(err.Error(), tt.wantErr):
t.Fatalf("Validate() = %v, want an error mentioning %q", err, tt.wantErr)
} }
}) })
} }
} }
func TestQueryValidateRequiresIdentity(t *testing.T) { func TestScopeKeyIsCanonical(t *testing.T) {
t.Parallel() t.Parallel()
if err := (Query{}).Validate(); err == nil { tests := []struct {
t.Fatal("Validate() = nil for empty identity, want error") name string
scope Scope
want string
}{
{"global", Scope{Kind: ScopeGlobal}, "global"},
{"agent", Scope{Kind: ScopeAgent, Agent: "telegram"}, "agent:8:telegram"},
{"user", Scope{Kind: ScopeUser, User: "u1"}, "user:2:u1"},
{"agent-user", Scope{Kind: ScopeAgentUser, Agent: "a1", User: "u1"}, "agent-user:2:a1:2:u1"},
// An invalid scope has no storage key; "" is the signal, never a key
// a valid scope could produce.
{"invalid kind", Scope{Kind: "session", Agent: "a1"}, ""},
{"missing component", Scope{Kind: ScopeAgentUser, Agent: "a1"}, "agent-user:2:a1:0:"},
} }
if err := (Query{Identity: "u1"}).Validate(); err != nil {
t.Fatalf("Validate() = %v for identity-only query, want nil (Text may be empty)", err) for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
if got := tt.scope.Key(); got != tt.want {
t.Fatalf("Key() = %q, want %q", got, tt.want)
}
})
}
}
// TestScopeKeyCollisionFree is the property Key's length prefix exists for:
// AgentID and IdentityID are channel-native and opaque, so a separator-based
// key would let two different scopes share one storage key. Every scope over
// ids that embed the separator must still key distinctly.
func TestScopeKeyCollisionFree(t *testing.T) {
t.Parallel()
ids := []string{"", "a", "a:", ":a", "a:1", "1", "1:a", "agent-user:1:a"}
distinct := map[Scope]bool{{Kind: ScopeGlobal}: true}
for _, agent := range ids {
for _, user := range ids {
distinct[Scope{Kind: ScopeAgentUser, Agent: AgentID(agent), User: IdentityID(user)}] = true
distinct[Scope{Kind: ScopeAgent, Agent: AgentID(agent)}] = true
distinct[Scope{Kind: ScopeUser, User: IdentityID(user)}] = true
}
}
seen := make(map[string]Scope, len(distinct))
for scope := range distinct {
if err := scope.Validate(); err != nil {
continue // an incomplete scope has no key to collide with
}
key := scope.Key()
if key == "" {
t.Fatalf("valid scope %+v has an empty key", scope)
}
if other, dup := seen[key]; dup {
t.Fatalf("scopes %+v and %+v share key %q", other, scope, key)
}
seen[key] = scope
}
}
func TestSubjectScopesNarrowestFirst(t *testing.T) {
t.Parallel()
tests := []struct {
name string
subject Subject
want []string
}{
{
name: "agent and user",
subject: Subject{AgentID: "a1", UserID: "u1"},
want: []string{"agent-user:2:a1:2:u1", "agent:2:a1", "user:2:u1", "global"},
},
{
// A channel with no per-person identity (dashboard, webhook)
// gets agent and global only -- never a wider scope to stand in
// for the one it does not know.
name: "agent only",
subject: Subject{AgentID: "a1"},
want: []string{"agent:2:a1", "global"},
},
{
name: "user only",
subject: Subject{UserID: "u1"},
want: []string{"user:2:u1", "global"},
},
{
name: "neither",
subject: Subject{},
want: []string{"global"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got := keys(tt.subject.Scopes())
if len(got) != len(tt.want) {
t.Fatalf("Scopes() = %v, want %v", got, tt.want)
}
for i := range got {
if got[i] != tt.want[i] {
t.Fatalf("Scopes() = %v, want %v", got, tt.want)
}
}
})
}
}
func TestSubjectWritableScopesExcludeGlobal(t *testing.T) {
t.Parallel()
subject := Subject{AgentID: "a1", UserID: "u1"}
writable := keys(subject.WritableScopes())
if len(writable) != 3 {
t.Fatalf("WritableScopes() = %v, want three scopes", writable)
}
for _, key := range writable {
if key == "global" {
t.Fatal("WritableScopes() includes global, which is never model-writable")
}
}
// Every writable scope is readable: the read set is a superset.
readable := map[string]bool{}
for _, key := range keys(subject.Scopes()) {
readable[key] = true
}
for _, key := range writable {
if !readable[key] {
t.Fatalf("writable scope %q is not in the read set", key)
}
}
}
func keys(scopes []Scope) []string {
out := make([]string, 0, len(scopes))
for _, scope := range scopes {
out = append(out, scope.Key())
}
return out
}
func TestNewRecordValidate(t *testing.T) {
t.Parallel()
valid := Scope{Kind: ScopeAgent, Agent: "a1"}
tests := []struct {
name string
in NewRecord
wantErr string
}{
{"valid", NewRecord{Scope: valid, Content: "x"}, ""},
{"valid without kind", NewRecord{Scope: valid, Content: "x"}, ""},
{"empty content", NewRecord{Scope: valid}, "content must not be empty"},
{"invalid scope", NewRecord{Scope: Scope{Kind: ScopeAgent}, Content: "x"}, "agent must not be empty"},
{"no scope", NewRecord{Content: "x"}, "unknown kind"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
assertValidation(t, tt.in.Validate(), tt.wantErr)
})
}
}
func TestRecordUpdateValidate(t *testing.T) {
t.Parallel()
valid := Scope{Kind: ScopeAgent, Agent: "a1"}
tests := []struct {
name string
in RecordUpdate
wantErr string
}{
{"valid", RecordUpdate{Scope: valid, ID: "r1", Content: "x"}, ""},
{"valid with expected", RecordUpdate{Scope: valid, ID: "r1", Content: "x", Expected: 2}, ""},
{"empty id", RecordUpdate{Scope: valid, Content: "x"}, "id must not be empty"},
{"empty content", RecordUpdate{Scope: valid, ID: "r1"}, "content must not be empty"},
{"invalid scope", RecordUpdate{Scope: Scope{Kind: ScopeGlobal, Agent: "a1"}, ID: "r1", Content: "x"}, "takes no agent or user"},
{"negative expected", RecordUpdate{Scope: valid, ID: "r1", Content: "x", Expected: -1}, "must not be negative"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
assertValidation(t, tt.in.Validate(), tt.wantErr)
})
}
}
func TestQueryValidate(t *testing.T) {
t.Parallel()
agent := Scope{Kind: ScopeAgent, Agent: "a1"}
tests := []struct {
name string
q Query
wantErr string
}{
{"one scope", Query{Scopes: []Scope{agent}}, ""},
{"two scopes", Query{Scopes: []Scope{agent, {Kind: ScopeGlobal}}}, ""},
{"scope only, no text", Query{Scopes: []Scope{agent}}, ""},
{"with limit", Query{Scopes: []Scope{agent}, Limit: 5}, ""},
{"no scopes", Query{}, "at least one scope"},
{"invalid scope", Query{Scopes: []Scope{{Kind: ScopeAgentUser, Agent: "a1"}}}, "both required"},
{"negative limit", Query{Scopes: []Scope{agent}, Limit: -1}, "must not be negative"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
assertValidation(t, tt.q.Validate(), tt.wantErr)
})
} }
} }
@@ -44,3 +276,15 @@ func TestManifestValidateAcceptsZeroValue(t *testing.T) {
t.Fatalf("Validate() = %v, want nil: no field is required yet", err) t.Fatalf("Validate() = %v, want nil: no field is required yet", err)
} }
} }
func assertValidation(t *testing.T, err error, wantErr string) {
t.Helper()
switch {
case wantErr == "" && err != nil:
t.Fatalf("Validate() = %v, want nil", err)
case wantErr != "" && err == nil:
t.Fatalf("Validate() = nil, want an error mentioning %q", wantErr)
case wantErr != "" && !strings.Contains(err.Error(), wantErr):
t.Fatalf("Validate() = %v, want an error mentioning %q", err, wantErr)
}
}
+11 -1
View File
@@ -1,6 +1,9 @@
package memory package memory
import "time" import (
"log/slog"
"time"
)
// Registrar is the narrow typed host access an engine receives at // Registrar is the narrow typed host access an engine receives at
// registration. Fields are typed contracts declared here — this package // registration. Fields are typed contracts declared here — this package
@@ -16,6 +19,13 @@ type Registrar struct {
// Clock is injectable time. Always bound; nil at construction is // Clock is injectable time. Always bound; nil at construction is
// replaced with the system clock. // replaced with the system clock.
Clock Clock Clock Clock
// Log is the engine-scoped diagnostic logger, optional so domain tests
// and embedders can construct a registrar without a logging sink (the
// curator family's Registrar carries the same field for the same
// reason). An engine emits here what it cannot emit as an event: the
// content scanner's warn level, which allows the write and is expected
// to leave a record of what it saw.
Log *slog.Logger
} }
// EventSink publishes engine activity. Mirrors curator.EventSink: emission // EventSink publishes engine activity. Mirrors curator.EventSink: emission
+82 -12
View File
@@ -4,6 +4,8 @@ import (
"context" "context"
"errors" "errors"
"slices" "slices"
"strconv"
"strings"
"sync" "sync"
"testing" "testing"
"time" "time"
@@ -21,12 +23,18 @@ type fakeEngine struct {
bindPanic bool bindPanic bool
view Registrar view Registrar
records map[string]Record records map[RecordID]Record
nextID int revisions map[RecordID][]Revision
nextID int
} }
func newFake(name string, manifest Manifest) *fakeEngine { func newFake(name string, manifest Manifest) *fakeEngine {
return &fakeEngine{name: name, manifest: manifest, records: map[string]Record{}} return &fakeEngine{
name: name,
manifest: manifest,
records: map[RecordID]Record{},
revisions: map[RecordID][]Revision{},
}
} }
func (f *fakeEngine) record(e string) { func (f *fakeEngine) record(e string) {
@@ -46,33 +54,95 @@ func (f *fakeEngine) Bind(host Registrar) {
} }
} }
func (f *fakeEngine) Write(_ context.Context, obs Observation) (Record, error) { // The fake implements the full Store so it satisfies MemoryEngine, but these
// tests exercise registration and lifecycle, not storage semantics -- the
// builtin engine's own suite is the store's real coverage. The in-memory
// behaviour here is deliberately minimal and not a reference implementation.
func (f *fakeEngine) Create(_ context.Context, in NewRecord) (Record, error) {
f.nextID++ f.nextID++
id := obs.Identity + "-" + time.Now().Format("150405.000000000") + "-" + string(rune('a'+f.nextID)) rec := Record{
rec := Record{ID: id, Identity: obs.Identity, Kind: obs.Kind, Content: obs.Content, At: obs.At, Metadata: obs.Metadata} ID: RecordID(f.name + "-" + strconv.Itoa(f.nextID)),
f.records[id] = rec Scope: in.Scope,
Kind: in.Kind,
Content: in.Content,
Revision: 1,
Author: in.Author,
OriginUser: in.OriginUser,
Source: in.Source,
}
f.records[rec.ID] = rec
return rec, nil
}
func (f *fakeEngine) Get(_ context.Context, scope Scope, id RecordID) (Record, error) {
rec, ok := f.records[id]
if !ok || rec.Scope != scope {
return Record{}, ErrNotFound
}
return rec, nil return rec, nil
} }
func (f *fakeEngine) Query(_ context.Context, q Query) ([]Record, error) { func (f *fakeEngine) Query(_ context.Context, q Query) ([]Record, error) {
wanted := make(map[Scope]bool, len(q.Scopes))
for _, scope := range q.Scopes {
wanted[scope] = true
}
var out []Record var out []Record
for _, r := range f.records { for _, r := range f.records {
if r.Identity == q.Identity { if q.Text != "" && !strings.Contains(r.Content, q.Text) {
continue
}
if wanted[r.Scope] {
out = append(out, r) out = append(out, r)
} }
} }
return out, nil slices.SortFunc(out, func(a, b Record) int { return strings.Compare(string(a.ID), string(b.ID)) })
return limit(out, q.Limit), nil
} }
func (f *fakeEngine) List(ctx context.Context, identity string) ([]Record, error) { func (f *fakeEngine) List(ctx context.Context, scope Scope) ([]Record, error) {
return f.Query(ctx, Query{Identity: identity}) return f.Query(ctx, Query{Scopes: []Scope{scope}})
} }
func (f *fakeEngine) Forget(_ context.Context, id string) error { func (f *fakeEngine) Update(_ context.Context, in RecordUpdate) (Record, error) {
rec, ok := f.records[in.ID]
if !ok || rec.Scope != in.Scope {
return Record{}, ErrNotFound
}
if in.Expected != 0 && in.Expected != rec.Revision {
return Record{}, ErrStaleRevision
}
f.revisions[rec.ID] = append(f.revisions[rec.ID], Revision{Record: rec, Deleted: false})
rec.Content = in.Content
rec.Revision++
rec.Author = in.Author
rec.Source = in.Source
f.records[rec.ID] = rec
return rec, nil
}
func (f *fakeEngine) Forget(_ context.Context, scope Scope, id RecordID) error {
rec, ok := f.records[id]
if !ok || rec.Scope != scope {
return nil
}
f.revisions[id] = append(f.revisions[id], Revision{Record: rec, Deleted: true})
delete(f.records, id) delete(f.records, id)
return nil return nil
} }
func (f *fakeEngine) Revisions(_ context.Context, _ Scope, id RecordID) ([]Revision, error) {
return f.revisions[id], nil
}
// limit trims out to n when n > 0, matching the Store contract's Limit.
func limit(out []Record, n int) []Record {
if n > 0 && len(out) > n {
return out[:n]
}
return out
}
func (f *fakeEngine) Start(context.Context) error { func (f *fakeEngine) Start(context.Context) error {
if f.startErr != nil { if f.startErr != nil {
return f.startErr return f.startErr
+123
View File
@@ -0,0 +1,123 @@
package memory
import "regexp"
// ThreatLevel indicates the severity of a threat detected by a Scanner.
type ThreatLevel int
const (
// ThreatNone means no suspicious patterns were found.
ThreatNone ThreatLevel = iota
// ThreatWarn means a potentially problematic pattern was found, but the
// write should be allowed with a warning logged.
ThreatWarn
// ThreatBlock means a dangerous pattern was found and the write should
// be rejected.
ThreatBlock
)
// ThreatResult describes a single threat detected during content scanning.
type ThreatResult struct {
// Level indicates the severity: none, warn, or block.
Level ThreatLevel
// Pattern is the regexp that matched (for audit logging).
Pattern string
// Message is a human-readable description of the threat.
Message string
}
// Scanner inspects memory content for dangerous or sensitive patterns before
// the content is persisted. Implementations are pluggable; the zero-value
// DefaultScanner is a reasonable default for production use.
type Scanner interface {
// ScanContent examines content and returns a ThreatResult. A result
// with Level == ThreatNone means the content is clean.
ScanContent(content string) ThreatResult
}
// DefaultScanner is the built-in Scanner implementation. It checks for
// prompt-injection patterns (block-level) and sensitive-data leakage
// (warn-level). The zero value is usable.
type DefaultScanner struct{}
// Compile-time check: DefaultScanner implements Scanner.
var _ Scanner = (*DefaultScanner)(nil)
// ── Patterns ────────────────────────────────────────────────────────────
//
// These patterns are compiled once at init time. They are intentionally
// broad to catch common variants without being so broad that they
// produce false positives on normal memory content.
var (
promptInjectionPatterns = []*regexp.Regexp{
// Override / ignore previous instructions
regexp.MustCompile(`(?i)(ignore|forget|override|disregard)\s+(all\s+)?(the\s+)?(previous|above|prior|earlier)\s+(instructions?|directions?|commands?|prompts?)`),
// Role-change / identity attacks
regexp.MustCompile(`(?i)(you\s+(are|now)\s+|act\s+as\s+|pretend\s+(to\s+be|you\s+are)\s+|roleplay\s+as\s+)(now\s+)?(an?\s+)?(different|new|another|unrestricted|uncensored)`),
// Delimiter injection (common in adversarial prompts)
regexp.MustCompile(`(?i)<\|im_start\|>`),
regexp.MustCompile(`(?i)<\|im_end\|>`),
// System prompt extraction attempts
regexp.MustCompile(`(?i)(print|output|repeat|display|show|tell\s+me)\s+(your\s+)?(system\s+)?(prompt|instructions?|directions?)\s*(exactly|verbatim)?`),
}
sensitiveDataPatterns = []*regexp.Regexp{
// API keys and secrets with common key=value patterns
regexp.MustCompile(`(?i)(api[_\s-]?key|apikey|api[_\s-]?secret)\s*[:=]\s*['"\x60]?[\w-]{8,}['"\x60]?`),
// Passwords in key=value form
regexp.MustCompile(`(?i)(password|passwd|pwd)\s*[:=]\s*['"\x60]?\S{4,}['"\x60]?`),
// Generic secrets and tokens
regexp.MustCompile(`(?i)(secret|token|credential|authorization)\s*[:=]\s*['"\x60]?[\w-]{8,}['"\x60]?`),
// Private key blocks (PEM format)
regexp.MustCompile(`(?i)-----BEGIN\s+(RSA\s+|EC\s+|DSA\s+|OPENSSH\s+)?PRIVATE\s+KEY-----`),
// JWT tokens (base64-encoded header.payload.signature pattern)
regexp.MustCompile(`eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}`),
}
)
// ScanContent examines content against the prompt-injection and
// sensitive-data patterns. Prompt-injection matches produce ThreatBlock;
// sensitive-data matches produce ThreatWarn. The first match wins.
func (s *DefaultScanner) ScanContent(content string) ThreatResult {
for _, p := range promptInjectionPatterns {
if m := p.FindString(content); m != "" {
return ThreatResult{
Level: ThreatBlock,
Pattern: p.String(),
Message: "content matches prompt injection pattern: " + truncateMatch(m),
}
}
}
for _, p := range sensitiveDataPatterns {
if m := p.FindString(content); m != "" {
return ThreatResult{
Level: ThreatWarn,
Pattern: p.String(),
Message: "content may contain sensitive data: " + truncateMatch(m),
}
}
}
return ThreatResult{Level: ThreatNone}
}
// truncateMatch limits the matched string for log/error messages so that
// a long match (e.g. an entire private key) is not echoed in full.
func truncateMatch(s string) string {
const maxMatchLen = 40
if len(s) <= maxMatchLen {
return s
}
return s[:maxMatchLen] + "..."
}
+429 -155
View File
@@ -1,6 +1,8 @@
// Package memory implements domain/memory.MemoryEngine against concrete // Package memory implements domain/memory.MemoryEngine over concrete
// backends. See docs/architecture/migration-decisions.md#5-memory-placement-and-storage // backends. The builtin engine persists each scope's records as markdown
// for the record/identity model this package's builtin engine follows. // under its root -- human-readable and hand-editable -- with a per-scope
// HISTORY.md retaining the states that are no longer live. See
// docs/prds/memory-engine-unification.md for the decision this implements.
package memory package memory
import ( import (
@@ -9,56 +11,71 @@ import (
"encoding/hex" "encoding/hex"
"errors" "errors"
"fmt" "fmt"
"log/slog"
"os" "os"
"path/filepath" "path/filepath"
"slices"
"strings" "strings"
"sync" "sync"
"time"
"github.com/google/uuid" "github.com/google/uuid"
domainmemory "github.com/samcharles93/archie-core/internal/domain/memory" domainmemory "github.com/samcharles93/archie-core/internal/domain/memory"
"github.com/samcharles93/archie-core/internal/memory/builtin" "github.com/samcharles93/archie-core/internal/infrastructure/memory/builtin"
) )
// EngineName is the name BuiltinEngine registers under and the default the // EngineName is the name BuiltinEngine registers under and the default the
// config surface (archie-core-1786637490043) resolves to when no engine is // config surface resolves to when no engine is configured.
// configured.
const EngineName = "builtin" const EngineName = "builtin"
// markerPrefix opens the leading line every block written through this // defaultSectionName is the section a record with an empty Kind is written
// engine carries, embedding the record's stable ID. See the migration // to.
// decision this package implements: an ID assigned once at Write, never
// derived from content, so a later edit elsewhere in the file can never
// orphan a Forget.
const (
markerPrefix = "<!--mem:"
markerSuffix = "-->"
)
// defaultSectionName is the section an Observation with an empty Kind is
// written to.
const defaultSectionName = "general" const defaultSectionName = "general"
// BuiltinEngine implements domain/memory.MemoryEngine over one // liveFileName is the one live markdown document per scope, holding that
// internal/memory/builtin.Store per identity. internal/memory/builtin is // scope's heads.
// unchanged by this package -- see the migration decision this const liveFileName = "OBSERVATIONS.md"
// implements for why.
// defaultQueryLimit bounds Query when the caller names no limit: 0 in
// Query.Limit means "the engine's default", never unlimited, and List is the
// unbounded read for one scope.
const defaultQueryLimit = 100
// contentScanner is the family's content scanner, applied in Create and
// Update. Those two are the one choke point every producer crosses, so the
// curator's model-extracted content is covered as well as the chat tool's
// (docs/prds/memory-engine-unification.md §7).
var contentScanner domainmemory.Scanner = &domainmemory.DefaultScanner{}
// scopeStores is one scope's on-disk pair: the heads, and the states that
// used to be heads.
type scopeStores struct {
live *builtin.Store
history *builtin.Store
}
// BuiltinEngine implements domain/memory.MemoryEngine over one pair of
// builtin.Store documents per scope.
type BuiltinEngine struct { type BuiltinEngine struct {
root string root string
maxFileBytes int maxFileBytes int
mu sync.Mutex mu sync.Mutex
stores map[string]*builtin.Store // keyed by identity, lazily created stores map[string]*scopeStores // keyed by scope key, lazily created
clock domainmemory.Clock
log *slog.Logger
} }
// NewBuiltinEngine builds an engine that persists each identity's // NewBuiltinEngine builds an engine that persists each scope under its own
// observations under its own subdirectory of root, named // subdirectory of root, named hex(sha256(scope.Key())): never the key or an
// hex(sha256(identity)) -- never the identity string itself, so an // id as a path component, so an opaque channel id can never traverse out of
// arbitrary identity can never traverse outside root and two identities // root and two ids a filesystem treats as equivalent can never collide.
// differing only in filesystem-equivalent characters can never collide. // maxFileBytes <= 0 uses the store's own default for the live document;
// maxFileBytes <= 0 uses the store's own default. // HISTORY.md always uses historyMaxFileBytes.
func NewBuiltinEngine(root string, maxFileBytes int) *BuiltinEngine { func NewBuiltinEngine(root string, maxFileBytes int) *BuiltinEngine {
return &BuiltinEngine{root: root, maxFileBytes: maxFileBytes, stores: make(map[string]*builtin.Store)} return &BuiltinEngine{root: root, maxFileBytes: maxFileBytes, stores: make(map[string]*scopeStores)}
} }
func (e *BuiltinEngine) Name() string { return EngineName } func (e *BuiltinEngine) Name() string { return EngineName }
@@ -68,7 +85,21 @@ func (e *BuiltinEngine) Manifest() domainmemory.Manifest {
return domainmemory.Manifest{RequiresNetwork: false} return domainmemory.Manifest{RequiresNetwork: false}
} }
func (e *BuiltinEngine) Bind(domainmemory.Registrar) {} // Bind takes the registrar's clock, which stamps CreatedAt/UpdatedAt, and its
// optional logger.
//
// It deliberately ignores Registrar.Events. The only warn-level finding this
// engine produces is a scanner hit, and there is no ratified memory event
// vocabulary to emit it in: internal/events declares kinds for tasks,
// curators, workflow stages and scheduling, none for memory, and the
// composition binds no memory event sink at all (bootstrap registers the
// engine with an empty Registrar). Emitting an invented kind into a sink
// nobody binds would look wired and not be. The scanner's warning is a
// diagnostic, not an event, and goes to Registrar.Log (see scanContent).
func (e *BuiltinEngine) Bind(host domainmemory.Registrar) {
e.clock = host.Clock
e.log = host.Log
}
func (e *BuiltinEngine) Start(context.Context) error { return nil } func (e *BuiltinEngine) Start(context.Context) error { return nil }
@@ -78,37 +109,77 @@ func (e *BuiltinEngine) Health(context.Context) domainmemory.Health {
func (e *BuiltinEngine) Stop(context.Context) error { return nil } func (e *BuiltinEngine) Stop(context.Context) error { return nil }
// storeFor returns the identity's store, creating and caching it on first // systemClock is the fallback for an engine no registry bound. Registrar's
// use. // own doc says a nil Clock is replaced at construction, but a directly
func (e *BuiltinEngine) storeFor(identity string) (*builtin.Store, error) { // constructed engine (a test, an embedder) has no registry at all, and a
e.mu.Lock() // clock nil-panic on a write would be a poor way to find that out.
defer e.mu.Unlock() type systemClock struct{}
if s, ok := e.stores[identity]; ok {
return s, nil func (systemClock) Now() time.Time { return time.Now() }
var fallbackClock domainmemory.Clock = systemClock{}
func (e *BuiltinEngine) now() time.Time {
if e.clock == nil {
return fallbackClock.Now()
} }
dir := filepath.Join(e.root, identityDir(identity)) return e.clock.Now()
if err := os.MkdirAll(dir, 0o755); err != nil {
return nil, fmt.Errorf("memory: builtin engine: identity directory %s unusable: %w", dir, err)
}
s, err := builtin.NewStore(filepath.Join(dir, "OBSERVATIONS.md"), e.maxFileBytes)
if err != nil {
return nil, err
}
e.stores[identity] = s
return s, nil
} }
// identityDir maps an identity to a filesystem-safe, collision-resistant // ── storage ────────────────────────────────────────────────────────────
// scopeStoresFor returns the scope's document pair, opening it on first use
// and caching it. create is false on every read path: a read must not leave
// a scope directory behind, and a scope nothing was ever written to answers
// as absent rather than as empty-but-created.
//
// Callers must have validated the scope already: Scope.Key() is empty for an
// invalid scope, and caching on "" would pool every invalid scope into one
// store under one directory.
func (e *BuiltinEngine) scopeStoresFor(scope domainmemory.Scope, create bool) (*scopeStores, bool, error) {
key := scope.Key()
e.mu.Lock()
defer e.mu.Unlock()
if s, ok := e.stores[key]; ok {
return s, true, nil
}
dir := filepath.Join(e.root, scopeDirName(scope))
if create {
if err := os.MkdirAll(dir, 0o755); err != nil {
return nil, false, fmt.Errorf("memory: builtin engine: scope directory %s unusable: %w", dir, err)
}
} else if _, err := os.Stat(dir); err != nil {
if os.IsNotExist(err) {
return nil, false, nil
}
return nil, false, fmt.Errorf("memory: builtin engine: scope directory %s unusable: %w", dir, err)
}
live, err := builtin.NewStore(filepath.Join(dir, liveFileName), e.maxFileBytes)
if err != nil {
return nil, false, err
}
history, err := builtin.NewStore(filepath.Join(dir, historyFileName), historyMaxFileBytes)
if err != nil {
return nil, false, err
}
stores := &scopeStores{live: live, history: history}
e.stores[key] = stores
return stores, true, nil
}
// scopeDirName maps a scope to a filesystem-safe, collision-resistant
// directory name. // directory name.
func identityDir(identity string) string { func scopeDirName(scope domainmemory.Scope) string {
sum := sha256.Sum256([]byte(identity)) sum := sha256.Sum256([]byte(scope.Key()))
return hex.EncodeToString(sum[:]) return hex.EncodeToString(sum[:])
} }
// sectionFor returns the section an Observation's Kind maps to, and // sectionFor returns the section a record's Kind maps to, validated the same
// validates it the same way builtin already validates any section name -- // way builtin validates any section name -- so an invalid Kind fails with the
// so an invalid Kind fails with the same error shape memory_edit already // error shape the memory tool already returns for an invalid section, not a
// returns for an invalid section, not a new one. // new one.
func sectionFor(kind string) (string, error) { func sectionFor(kind string) (string, error) {
if kind == "" { if kind == "" {
kind = defaultSectionName kind = defaultSectionName
@@ -119,111 +190,314 @@ func sectionFor(kind string) (string, error) {
return kind, nil return kind, nil
} }
func (e *BuiltinEngine) Write(_ context.Context, obs domainmemory.Observation) (domainmemory.Record, error) { // scanContent applies the family's scanner to content about to be persisted,
if err := obs.Validate(); err != nil { // before anything is written.
return domainmemory.Record{}, err //
} // A block-level threat fails the write loudly -- loudly, because a scanner
section, err := sectionFor(obs.Kind) // hit that only logs is a control that looks wired and is not -- while a
if err != nil { // warn-level hit is stored and logged, which is what its level is documented
return domainmemory.Record{}, err // to mean ("allow the write but emit a warning"). Warn covers the
} // sensitive-data patterns: an API key, a token or a private key in stored
s, err := e.storeFor(obs.Identity) // content is worth an audit line even though it is not worth refusing the
if err != nil { // write over, and a warning that goes nowhere is the same as no scan at all.
return domainmemory.Record{}, err func (e *BuiltinEngine) scanContent(content string) error {
} result := contentScanner.ScanContent(content)
switch result.Level {
id := uuid.NewString() case domainmemory.ThreatBlock:
marked := markerPrefix + id + markerSuffix + "\n" + obs.Content return fmt.Errorf("memory: builtin engine: refusing to persist content: %s", result.Message)
if _, err := s.Add(section, marked); err != nil { case domainmemory.ThreatWarn:
return domainmemory.Record{}, err if e.log != nil {
} e.log.Warn("memory: stored content matched a sensitive-data pattern",
"pattern", result.Pattern, "reason", result.Message)
return domainmemory.Record{
ID: id,
Identity: obs.Identity,
Kind: section,
Content: obs.Content,
At: obs.At,
Metadata: obs.Metadata,
}, nil
}
func (e *BuiltinEngine) Query(ctx context.Context, q domainmemory.Query) ([]domainmemory.Record, error) {
if err := q.Validate(); err != nil {
return nil, err
}
all, err := e.List(ctx, q.Identity)
if err != nil {
return nil, err
}
if q.Text == "" {
return limitRecords(all, q.Limit), nil
}
var matched []domainmemory.Record
for _, r := range all {
if strings.Contains(r.Content, q.Text) {
matched = append(matched, r)
} }
} }
return limitRecords(matched, q.Limit), nil
}
func limitRecords(records []domainmemory.Record, limit int) []domainmemory.Record {
if limit > 0 && len(records) > limit {
return records[:limit]
}
return records
}
func (e *BuiltinEngine) List(_ context.Context, identity string) ([]domainmemory.Record, error) {
if identity == "" {
return nil, errors.New("memory: builtin engine: identity must not be empty")
}
s, err := e.storeFor(identity)
if err != nil {
return nil, err
}
return parseRecords(identity, s.Render()), nil
}
func (e *BuiltinEngine) Forget(_ context.Context, id string) error {
if id == "" {
return errors.New("memory: builtin engine: id must not be empty")
}
marker := markerPrefix + id + markerSuffix
e.mu.Lock()
stores := make([]*builtin.Store, 0, len(e.stores))
for _, s := range e.stores {
stores = append(stores, s)
}
e.mu.Unlock()
// Forgetting an id that does not exist is not an error (contract.go's
// Forget doc): the end state -- id absent -- already holds whether or
// not any store ever had it, so checking every known identity's store
// and returning nil regardless of a miss is correct, not a fallback
// masking a real failure.
for _, s := range stores {
section, ok := sectionOf(s.Render(), id)
if !ok {
continue
}
if _, err := s.Remove(section, marker, true); err != nil {
return err
}
return nil
}
return nil return nil
} }
// sectionOf finds which section of rendered carries a block whose marker // storeContent returns the content a block will actually hold, so that the
// names id, so Forget can call Store.Remove with the section it actually // record Create and Update answer with is the record a later Get reads back
// requires without a mutating probe. ok is false if no block does. // rather than what the caller handed in.
func sectionOf(rendered, id string) (section string, ok bool) { //
for _, rec := range parseRecords("", rendered) { // The trimming is what makes that true at the content's edges: builtin.Store
if rec.ID == id { // trims a block, so content that begins or ends with whitespace would come
return rec.Kind, true // back without it -- or, for whitespace that is itself a blank line, as an
// empty record. Content that is nothing but whitespace is rejected here for
// the same reason. Interior blank lines and "## " lines are the block
// format's own delimiters rather than whitespace at an edge, and are carried
// through instead: renderBlock escapes them and parseBlocks undoes it, so
// accepted content is never cut at one.
func storeContent(content string) (string, error) {
trimmed := strings.TrimSpace(content)
if trimmed == "" {
return "", errors.New("memory: builtin engine: content must not be empty")
}
return trimmed, nil
}
// findBlock returns the live block addressed by id.
func findBlock(store *builtin.Store, id domainmemory.RecordID) (parsedBlock, bool) {
for _, block := range parseBlocks(store.Render()) {
if block.marker.ID == string(id) {
return block, true
} }
} }
return "", false return parsedBlock{}, false
}
func notFound(scope domainmemory.Scope, id domainmemory.RecordID) error {
return fmt.Errorf("memory: builtin engine: %w: id %q in scope %s", domainmemory.ErrNotFound, id, scope)
}
// ── Store ──────────────────────────────────────────────────────────────
// Create records one new memory: the id is assigned once, here, and is the
// only address the record ever has.
func (e *BuiltinEngine) Create(_ context.Context, in domainmemory.NewRecord) (domainmemory.Record, error) {
if err := in.Validate(); err != nil {
return domainmemory.Record{}, err
}
section, err := sectionFor(in.Kind)
if err != nil {
return domainmemory.Record{}, err
}
if err := e.scanContent(in.Content); err != nil {
return domainmemory.Record{}, err
}
content, err := storeContent(in.Content)
if err != nil {
return domainmemory.Record{}, err
}
stores, _, err := e.scopeStoresFor(in.Scope, true)
if err != nil {
return domainmemory.Record{}, err
}
now := e.now()
marker := markerData{
ID: uuid.NewString(),
Revision: 1,
Kind: section,
Author: in.Author,
OriginUser: string(in.OriginUser),
Source: in.Source,
CreatedAt: now,
UpdatedAt: now,
}
block, err := renderBlock(marker, content)
if err != nil {
return domainmemory.Record{}, err
}
if _, err := stores.live.Add(section, block); err != nil {
return domainmemory.Record{}, err
}
return recordOf(marker, in.Scope, content), nil
}
func (e *BuiltinEngine) Get(_ context.Context, scope domainmemory.Scope, id domainmemory.RecordID) (domainmemory.Record, error) {
if err := scope.Validate(); err != nil {
return domainmemory.Record{}, err
}
if id == "" {
return domainmemory.Record{}, errors.New("memory: builtin engine: id must not be empty")
}
stores, ok, err := e.scopeStoresFor(scope, false)
if err != nil {
return domainmemory.Record{}, err
}
if !ok {
return domainmemory.Record{}, notFound(scope, id)
}
block, ok := findBlock(stores.live, id)
if !ok {
return domainmemory.Record{}, notFound(scope, id)
}
return block.record(scope), nil
}
// List returns one scope's heads, most recently updated first.
func (e *BuiltinEngine) List(_ context.Context, scope domainmemory.Scope) ([]domainmemory.Record, error) {
if err := scope.Validate(); err != nil {
return nil, err
}
stores, ok, err := e.scopeStoresFor(scope, false)
if err != nil {
return nil, err
}
if !ok {
return nil, nil
}
blocks := parseBlocks(stores.live.Render())
records := make([]domainmemory.Record, 0, len(blocks))
for _, block := range blocks {
records = append(records, block.record(scope))
}
return orderHeads(records), nil
}
// orderHeads returns heads most recently updated first. The input is in
// document order, which is creation order for a document this engine wrote;
// reversing before the stable sort makes creation order the tie-break for two
// records sharing an UpdatedAt, which is the common case for a coarse clock or
// one Create burst.
func orderHeads(records []domainmemory.Record) []domainmemory.Record {
slices.Reverse(records)
slices.SortStableFunc(records, func(a, b domainmemory.Record) int {
return b.UpdatedAt.Compare(a.UpdatedAt)
})
return records
}
// Query returns the heads of exactly the scopes q names -- the caller's read
// set, never widened here -- grouped in the order those scopes were given and
// bounded by q.Limit across the whole result.
func (e *BuiltinEngine) Query(ctx context.Context, q domainmemory.Query) ([]domainmemory.Record, error) {
if err := q.Validate(); err != nil {
return nil, err
}
limit := q.Limit
if limit == 0 {
limit = defaultQueryLimit
}
var out []domainmemory.Record
for _, scope := range q.Scopes {
heads, err := e.List(ctx, scope)
if err != nil {
return nil, err
}
for _, record := range heads {
if q.Text != "" && !strings.Contains(record.Content, q.Text) {
continue
}
out = append(out, record)
if len(out) == limit {
return out, nil
}
}
}
return out, nil
}
// Update supersedes a record's content in place and returns the new state.
//
// A record may not move between scopes, which is why RecordUpdate carries no
// scope to change: it is addressed by the scope it is already in.
func (e *BuiltinEngine) Update(_ context.Context, in domainmemory.RecordUpdate) (domainmemory.Record, error) {
if err := in.Validate(); err != nil {
return domainmemory.Record{}, err
}
// Scanned before anything is written: a refused update must leave both
// the record and its history untouched.
if err := e.scanContent(in.Content); err != nil {
return domainmemory.Record{}, err
}
content, err := storeContent(in.Content)
if err != nil {
return domainmemory.Record{}, err
}
stores, ok, err := e.scopeStoresFor(in.Scope, false)
if err != nil {
return domainmemory.Record{}, err
}
if !ok {
return domainmemory.Record{}, notFound(in.Scope, in.ID)
}
// Read the document as it is on disk before anything is decided from it.
// This engine caches the scope's document and rewrites it whole (the PRD
// leaves "two processes, one scope file" undecided), so the copy it holds
// can be a whole write out of date -- the Expected check below would
// compare against a revision that is already superseded, and the rewrite
// that follows it would write the stale copy back over the current one.
// What this does not fix: the rewrite is still last-writer-wins for this
// record and every other one, between the reload and the write.
if err := stores.live.Reload(); err != nil {
return domainmemory.Record{}, err
}
current, ok := findBlock(stores.live, in.ID)
if !ok {
return domainmemory.Record{}, notFound(in.Scope, in.ID)
}
// Expected is what makes a lost update detectable: the revision the
// caller last saw is the check, against the revision just read.
if in.Expected != 0 && in.Expected != current.marker.Revision {
return domainmemory.Record{}, fmt.Errorf(
"memory: builtin engine: %w: record %q is at revision %d, not %d",
domainmemory.ErrStaleRevision, in.ID, current.marker.Revision, in.Expected,
)
}
now := e.now()
next := current.marker
next.Revision = current.marker.Revision + 1
next.Author = in.Author
next.Source = in.Source
next.UpdatedAt = now
// Crash safety, in this order: the superseded state is durable in
// HISTORY.md before the live block is touched. A crash between the two
// leaves an extra history entry and an unchanged record -- recoverable,
// per §2 -- and never a record whose previous state is gone.
if err := appendHistory(stores.history, current.marker, current.content(), now, false); err != nil {
return domainmemory.Record{}, err
}
block, err := renderBlock(next, content)
if err != nil {
return domainmemory.Record{}, err
}
if _, err := stores.live.Replace(current.section, current.text, block, true); err != nil {
return domainmemory.Record{}, err
}
return recordOf(next, in.Scope, content), nil
}
// Forget removes a record's live state, retaining a deleted revision so its
// provenance outlives its content. Forgetting an id that is not there is not
// an error: the end state, id absent, already holds.
func (e *BuiltinEngine) Forget(_ context.Context, scope domainmemory.Scope, id domainmemory.RecordID) error {
if err := scope.Validate(); err != nil {
return err
}
if id == "" {
return errors.New("memory: builtin engine: id must not be empty")
}
stores, ok, err := e.scopeStoresFor(scope, false)
if err != nil {
return err
}
if !ok {
return nil
}
current, ok := findBlock(stores.live, id)
if !ok {
return nil
}
// The same crash-safe order as Update: the deleted revision is durable
// before the live block disappears, so the record's provenance is never
// the part that is lost.
if err := appendHistory(stores.history, current.marker, current.content(), e.now(), true); err != nil {
return err
}
_, err = stores.live.Remove(current.section, current.text, true)
return err
}
// Revisions returns a record's retained states, oldest first, including the
// deleted revision Forget records. A record that never existed is empty, not
// an error.
func (e *BuiltinEngine) Revisions(_ context.Context, scope domainmemory.Scope, id domainmemory.RecordID) ([]domainmemory.Revision, error) {
if err := scope.Validate(); err != nil {
return nil, err
}
if id == "" {
return nil, errors.New("memory: builtin engine: id must not be empty")
}
stores, ok, err := e.scopeStoresFor(scope, false)
if err != nil {
return nil, err
}
if !ok {
return nil, nil
}
return revisionsOf(stores.history, scope, id), nil
} }
File diff suppressed because it is too large. Load diff
+76
View File
@@ -0,0 +1,76 @@
package memory
import (
"fmt"
"time"
domainmemory "github.com/samcharles93/archie-core/internal/domain/memory"
"github.com/samcharles93/archie-core/internal/infrastructure/memory/builtin"
)
// historyFileName is the second document in each scope's directory: every
// state the scope's records no longer live in -- superseded by an Update, or
// removed by a Forget -- so a record's provenance outlives its content
// (docs/prds/memory-engine-unification.md §2).
const historyFileName = "HISTORY.md"
// historySectionName is the single section every retained state is appended
// to. One section is enough: each block's marker carries its record id, so
// grouping by kind would be a second index over the same file.
const historySectionName = "history"
// historyMaxFileBytes bounds each scope's HISTORY.md, deliberately far above
// the live document's bound (builtin's 100KB default, which is what
// production passes: bootstrap builds the engine with a 0 maxFileBytes).
//
// It has its own bound because history grows on every Update and every
// Forget while the live document does not. Inheriting the live document's
// bound would make a scope start refusing updates after a handful of
// revisions of one large record -- trading the PRD's open "unbounded
// history" question ("Not determined", no compaction policy) for a new
// failure mode at the point of write. That question stays open; this bound
// only makes reaching it a deliberate, enormous amount of history rather
// than an accident.
const historyMaxFileBytes = 4 * 1024 * 1024
// appendHistory appends one retained state -- the record as it was while
// live, plus when it stopped being live and whether it was deleted -- to the
// scope's history document. Content is passed beside m because the marker
// carries provenance, never the record's bytes.
func appendHistory(h *builtin.Store, m markerData, content string, at time.Time, deleted bool) error {
m.SupersededAt = &at
m.Deleted = deleted
block, err := renderBlock(m, content)
if err != nil {
return err
}
if _, err := h.Add(historySectionName, block); err != nil {
return fmt.Errorf("memory: builtin engine: retaining the superseded state of record %q: %w", m.ID, err)
}
return nil
}
// revisionsOf returns one record's retained states as HISTORY.md holds them,
// oldest first. Document order is append order, and a retained state is
// appended exactly when it stops being live, so no sort is needed -- or
// wanted: a crash between a history append and the live write leaves two
// entries for one revision (§2, recoverable), and sorting would not make
// that less true.
func revisionsOf(h *builtin.Store, scope domainmemory.Scope, id domainmemory.RecordID) []domainmemory.Revision {
var revisions []domainmemory.Revision
for _, block := range parseBlocks(h.Render()) {
if block.marker.ID != string(id) {
continue
}
var supersededAt time.Time
if block.marker.SupersededAt != nil {
supersededAt = *block.marker.SupersededAt
}
revisions = append(revisions, domainmemory.Revision{
Record: block.record(scope),
SupersededAt: supersededAt,
Deleted: block.marker.Deleted,
})
}
return revisions
}
+82
View File
@@ -0,0 +1,82 @@
package memory
import (
"bytes"
"encoding/json"
"fmt"
"strings"
"time"
)
// The engine writes each record as one blank-line-delimited markdown block
// whose first line is a marker comment followed by the record's content, so
// a block stays readable and editable by hand. The marker began life as a
// bare id (`<!--mem:<ulid>-->`, the incumbent format); it is JSON now
// because the retained provenance -- revision, kind, author, originating
// user, source and both timestamps -- has nowhere else to live, which is
// exactly what the PRD's Settles table records the bare form could not
// carry.
const (
markerPrefix = "<!--mem:"
markerSuffix = "-->"
)
// markerData is the provenance one block's marker carries. The same shape is
// written into a scope's live document and into its HISTORY.md; the last two
// fields are set only on a retained state, so a live block never carries
// them.
//
// json escapes newlines, so the marker is always exactly one line: a field
// containing one cannot break the block's first-line shape.
type markerData struct {
ID string `json:"id"`
Revision int `json:"revision"`
Kind string `json:"kind"`
Author string `json:"author,omitempty"`
OriginUser string `json:"origin_user,omitempty"`
Source string `json:"source,omitempty"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
// SupersededAt is when this state stopped being live. Set only on a
// state written to HISTORY.md.
SupersededAt *time.Time `json:"superseded_at,omitempty"`
// Deleted marks the state Forget retained: the record's last live
// content, kept so its provenance outlives it. Set only in HISTORY.md.
Deleted bool `json:"deleted,omitempty"`
}
// renderMarker encodes one marker line.
func renderMarker(m markerData) (string, error) {
var buf bytes.Buffer
enc := json.NewEncoder(&buf)
// The marker is documented as hand-editable, so it must not be
// HTML-escaped into \u003c runs an operator cannot read back. The suffix
// is still unambiguous either way: the body is cut at the line's end.
enc.SetEscapeHTML(false)
if err := enc.Encode(m); err != nil {
return "", fmt.Errorf("memory: builtin engine: encoding a record marker: %w", err)
}
return markerPrefix + strings.TrimRight(buf.String(), "\n") + markerSuffix, nil
}
// parseMarker decodes a block's first line. ok is false for a line that is
// not a marker, or one that names no record: a block without a usable marker
// is not one this engine ever wrote.
func parseMarker(line string) (markerData, bool) {
body, ok := strings.CutPrefix(line, markerPrefix)
if !ok {
return markerData{}, false
}
body, ok = strings.CutSuffix(body, markerSuffix)
if !ok {
return markerData{}, false
}
var m markerData
if err := json.Unmarshal([]byte(body), &m); err != nil {
return markerData{}, false
}
if m.ID == "" {
return markerData{}, false
}
return m, true
}
+128 -39
View File
@@ -6,31 +6,140 @@ import (
domainmemory "github.com/samcharles93/archie-core/internal/domain/memory" domainmemory "github.com/samcharles93/archie-core/internal/domain/memory"
) )
// sectionHeaderPrefix matches internal/memory/builtin's own format (its // sectionHeaderPrefix matches the markdown store's own format (builtin's
// sectionHeaderPrefix constant is unexported): a "## " line starts a named // sectionHeaderPrefix is unexported): a "## " line starts a named section,
// section, and blocks within it are separated by one blank line. This // and blocks within it are separated by one blank line. This package only
// package only ever reads what it wrote through builtin.Store.Add, so it // reads back what it wrote through builtin.Store.Add, so it only needs that
// only needs to read that one documented format back, not the general // one documented format, not the general case builtin.Store itself parses.
// case builtin.Store itself parses. //
// Those two delimiters are the format's structure, and builtin.Store owns
// them: a blank line ends a block and a "## " line starts a section, so
// content holding either would be read back as structure -- cut at the
// delimiter, the remainder unaddressable. Content is therefore escaped on
// the way out and unescaped on the way back (see encodeBlockContent), which
// keeps a record's content whole without giving the store a second format to
// parse.
const sectionHeaderPrefix = "## " const sectionHeaderPrefix = "## "
// parseRecords reads every marked block out of rendered and returns it as // parsedBlock is one marked block read back out of a rendered document: the
// a Record scoped to identity. A block with no marker was not written by // provenance its marker carried, the section it sat in, and its exact text
// this engine (e.g. hand-edited) and is skipped: this engine's contract is // within that document. Update replaces a block by that text, so it is never
// only ever asked about records it assigned an ID to. // reconstructed from fields and can never disagree with what is on disk.
func parseRecords(identity, rendered string) []domainmemory.Record { type parsedBlock struct {
var records []domainmemory.Record marker markerData
section string
text string
}
// content returns the block's content with the marker line stripped and the
// format's escaping undone.
func (b parsedBlock) content() string {
_, rest, _ := strings.Cut(b.text, "\n")
return decodeBlockContent(rest)
}
// record renders the block as a record of scope. Kind comes from the marker
// rather than the section: every record's retained states live in one
// "history" section, so the section cannot be the source of a kind.
func (b parsedBlock) record(scope domainmemory.Scope) domainmemory.Record {
return recordOf(b.marker, scope, b.content())
}
// record renders a marker and its content as a record of scope.
func recordOf(m markerData, scope domainmemory.Scope, content string) domainmemory.Record {
return domainmemory.Record{
ID: domainmemory.RecordID(m.ID),
Scope: scope,
Kind: m.Kind,
Content: content,
Revision: m.Revision,
Author: m.Author,
OriginUser: domainmemory.IdentityID(m.OriginUser),
Source: m.Source,
CreatedAt: m.CreatedAt,
UpdatedAt: m.UpdatedAt,
}
}
// renderBlock renders one record as the markdown block the store persists:
// the marker line, then the content.
func renderBlock(m markerData, content string) (string, error) {
line, err := renderMarker(m)
if err != nil {
return "", err
}
return line + "\n" + encodeBlockContent(content), nil
}
// contentEscape is the backslash that marks a content line the format would
// otherwise read as structure.
const contentEscape = `\`
// encodeBlockContent escapes the content lines the store's format would
// otherwise read as its own structure -- a blank line ends a block, and a
// "## " line starts a section -- so that a record's content survives the
// write whole (see sectionHeaderPrefix). An empty line becomes one backslash,
// and a line already starting with a backslash or with "## " gains one, so
// the two are never ambiguous: everything is written verbatim except lines
// that start with a backslash, and every content line can therefore be
// recovered by dropping the backslash it starts with (decodeBlockContent).
// The common record -- one paragraph, no delimiter in sight -- is written
// exactly as the caller wrote it, so the document stays hand-editable.
func encodeBlockContent(content string) string {
lines := strings.Split(content, "\n")
for i, line := range lines {
switch {
case line == "":
lines[i] = contentEscape
case strings.HasPrefix(line, contentEscape), strings.HasPrefix(line, sectionHeaderPrefix):
lines[i] = contentEscape + line
}
}
return strings.Join(lines, "\n")
}
// decodeBlockContent is encodeBlockContent's inverse: it returns the content
// the block's lines were rendered from.
func decodeBlockContent(text string) string {
lines := strings.Split(text, "\n")
for i, line := range lines {
if line == contentEscape {
lines[i] = ""
continue
}
lines[i] = strings.TrimPrefix(line, contentEscape)
}
return strings.Join(lines, "\n")
}
// parseBlocks reads every marked block out of a rendered document.
//
// A block with no marker, or one whose marker does not decode, was not
// written by this engine -- an operator hand-edited the file, or it predates
// the marker -- and is skipped. This engine's Store contract is only ever
// asked about records it assigned an id to, so inventing an id for an
// unmarked block would be a worse answer than not seeing it.
//
// Blocks are returned in document order, which is append order for a
// document this engine wrote, and therefore the order HISTORY.md's retained
// states come back in.
func parseBlocks(rendered string) []parsedBlock {
var blocks []parsedBlock
section := "" section := ""
var blockLines []string var lines []string
flush := func() { flush := func() {
if len(blockLines) == 0 { if len(lines) == 0 {
return return
} }
if rec, ok := parseBlock(identity, section, strings.Join(blockLines, "\n")); ok { text := strings.Join(lines, "\n")
records = append(records, rec) lines = nil
first, _, _ := strings.Cut(text, "\n")
marker, ok := parseMarker(first)
if !ok {
return
} }
blockLines = nil blocks = append(blocks, parsedBlock{marker: marker, section: section, text: text})
} }
for line := range strings.SplitSeq(rendered, "\n") { for line := range strings.SplitSeq(rendered, "\n") {
@@ -43,28 +152,8 @@ func parseRecords(identity, rendered string) []domainmemory.Record {
flush() flush()
continue continue
} }
blockLines = append(blockLines, line) lines = append(lines, line)
} }
flush() flush()
return records return blocks
}
// parseBlock splits one block back into its marker-carried ID and content.
// ok is false for a block with no recognizable marker.
func parseBlock(identity, section, block string) (domainmemory.Record, bool) {
first, rest, _ := strings.Cut(block, "\n")
id, ok := strings.CutPrefix(first, markerPrefix)
if !ok {
return domainmemory.Record{}, false
}
id, ok = strings.CutSuffix(id, markerSuffix)
if !ok {
return domainmemory.Record{}, false
}
return domainmemory.Record{
ID: id,
Identity: identity,
Kind: section,
Content: rest,
}, true
} }
@@ -18,6 +18,14 @@ const Name = "session-memory"
// written. // written.
const ActionExtracted = "memory.extracted" const ActionExtracted = "memory.extracted"
// ActionSkipped records a session the pass could not attribute to exactly
// one participant, so nothing was written for it. The reason names how many
// distinct senders were found (zero for a dashboard or webhook session,
// more than one for a group chat), and how many user messages carried no
// sender at all -- unknown provenance rules a session out on its own, beside
// any number of identified senders.
const ActionSkipped = "memory.skipped"
// DefaultInterval is the check-in cadence used when nothing more // DefaultInterval is the check-in cadence used when nothing more
// specific configures one, matching skillcurator.DefaultInterval's // specific configures one, matching skillcurator.DefaultInterval's
// convention (no per-curator interval config surface exists yet). // convention (no per-curator interval config surface exists yet).
@@ -103,7 +111,7 @@ func (c *Curator) Pass(ctx context.Context, in curator.PassInput) (curator.PassR
var actions []curator.Action var actions []curator.Action
for _, sess := range sessions { for _, sess := range sessions {
action, err := c.reviewOne(ctx, engine, sess.ID) action, err := c.reviewOne(ctx, engine, sess)
if err != nil { if err != nil {
return curator.PassResult{}, fmt.Errorf("session %q: %w", sess.ID, err) return curator.PassResult{}, fmt.Errorf("session %q: %w", sess.ID, err)
} }
@@ -126,11 +134,14 @@ func effectiveSince(since, now time.Time) time.Time {
} }
// reviewOne extracts observations from one session's recent messages and // reviewOne extracts observations from one session's recent messages and
// writes them through engine. Returns nil (no Action) when nothing was // writes them through engine as agent-user memory for the session's single
// extracted -- matching the skill curator's "a pass with nothing to // participant. Returns nil (no Action) when nothing was extracted --
// report is not an error." // matching the skill curator's "a pass with nothing to report is not an
func (c *Curator) reviewOne(ctx context.Context, engine domainmemory.MemoryEngine, sessionID string) (*curator.Action, error) { // error" -- or when the session carries no messages at all. A session that
msgs, err := c.host.Conversations.Messages(ctx, sessionID, messageTailSize) // cannot be attributed to one participant is skipped with an Action saying
// so.
func (c *Curator) reviewOne(ctx context.Context, engine domainmemory.MemoryEngine, sess curator.SessionSummary) (*curator.Action, error) {
msgs, err := c.host.Conversations.Messages(ctx, sess.ID, messageTailSize)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -138,6 +149,20 @@ func (c *Curator) reviewOne(ctx context.Context, engine domainmemory.MemoryEngin
return nil, nil return nil, nil
} }
// Resolve the participant before spending a model call: an
// unattributable session has nowhere addressable to write, so there is
// nothing a model response could add. ScopeAgentUser is addressed by
// both ids, and a guess would file one person's facts under another's.
participant, reason := sessionParticipant(msgs)
if participant == "" {
return &curator.Action{
At: c.host.Clock.Now(),
Type: ActionSkipped,
Detail: sess.ID,
Reason: reason,
}, nil
}
text, err := c.askModel(ctx, msgs) text, err := c.askModel(ctx, msgs)
if err != nil { if err != nil {
// A real model-call failure (provider down, timeout) is a Pass // A real model-call failure (provider down, timeout) is a Pass
@@ -159,12 +184,25 @@ func (c *Curator) reviewOne(ctx context.Context, engine domainmemory.MemoryEngin
facts = facts[:maxObservationsPerSession] facts = facts[:maxObservationsPerSession]
} }
// Agent-user is the only defensible home for a derived fact about a
// person: agent scope would leak user A's facts to user B through the
// same agent, user scope would pool them across agents, and the session
// id this used to write stops meaning anything when the session ends.
// ScopeAgentUser is also what a later chat turn reads back, which is
// the whole point of writing these at all.
scope := domainmemory.Scope{
Kind: domainmemory.ScopeAgentUser,
Agent: domainmemory.AgentID(sess.AgentID),
User: domainmemory.IdentityID(participant),
}
for _, fact := range facts { for _, fact := range facts {
if _, err := engine.Write(ctx, domainmemory.Observation{ if _, err := engine.Create(ctx, domainmemory.NewRecord{
Identity: sessionID, Scope: scope,
Kind: "note", Kind: "note",
Content: fact, Content: fact,
At: c.host.Clock.Now(), Author: Name,
OriginUser: domainmemory.IdentityID(participant),
Source: sess.ID,
}); err != nil { }); err != nil {
return nil, err return nil, err
} }
@@ -173,11 +211,48 @@ func (c *Curator) reviewOne(ctx context.Context, engine domainmemory.MemoryEngin
return &curator.Action{ return &curator.Action{
At: c.host.Clock.Now(), At: c.host.Clock.Now(),
Type: ActionExtracted, Type: ActionExtracted,
Detail: fmt.Sprintf("%s: wrote %d observation(s)", sessionID, len(facts)), Detail: fmt.Sprintf("%s: wrote %d observation(s)", sess.ID, len(facts)),
Reason: "session active since last pass", Reason: "session active since last pass",
}, nil }, nil
} }
// sessionParticipant returns the one participant a session's memory may be
// attributed to, or -- when there is none -- the reason there isn't.
//
// Only user-role messages carry a person: an assistant message carries the
// bot, so it can never manufacture a participant.
//
// A user-role message with an empty SenderID is part of the excerpt the
// model reads but names nobody, so the session is unattributable outright,
// even beside exactly one identified sender. The alternative -- skipping it
// and attributing the rest -- would credit whoever that sender is with
// facts drawn from content of unknown provenance, which is the isolation
// failure the agent-user scope exists to prevent.
func sessionParticipant(msgs []curator.ConversationMessage) (participant, reason string) {
senders := make(map[string]struct{})
unidentified := 0
for _, m := range msgs {
if m.Role != "user" {
continue
}
if m.SenderID == "" {
unidentified++
continue
}
senders[m.SenderID] = struct{}{}
}
if unidentified > 0 {
return "", fmt.Sprintf("no single participant: %d distinct sender(s), %d unidentified user message(s)", len(senders), unidentified)
}
if len(senders) == 1 {
for id := range senders {
return id, ""
}
}
return "", fmt.Sprintf("no single participant: %d distinct sender(s)", len(senders))
}
// extractPrompt instructs the model to return a JSON array of short, // extractPrompt instructs the model to return a JSON array of short,
// durable strings -- facts, preferences, or corrections worth keeping -- // durable strings -- facts, preferences, or corrections worth keeping --
// or [] if nothing in the excerpt is worth keeping. No prose outside the // or [] if nothing in the excerpt is worth keeping. No prose outside the
@@ -3,6 +3,7 @@ package sessioncurator
import ( import (
"context" "context"
"errors" "errors"
"fmt"
"testing" "testing"
"time" "time"
@@ -14,6 +15,7 @@ import (
type fakeConversations struct { type fakeConversations struct {
sessions map[string]time.Time // id -> last active sessions map[string]time.Time // id -> last active
agents map[string]string // id -> the agent that served it
messages map[string][]curator.ConversationMessage messages map[string][]curator.ConversationMessage
} }
@@ -23,7 +25,7 @@ func (f *fakeConversations) RecentSessions(_ context.Context, since time.Time) (
if at.Before(since) { if at.Before(since) {
continue continue
} }
out = append(out, curator.SessionSummary{ID: id, LastActive: at}) out = append(out, curator.SessionSummary{ID: id, AgentID: f.agents[id], LastActive: at})
} }
return out, nil return out, nil
} }
@@ -36,6 +38,17 @@ func (f *fakeConversations) Messages(_ context.Context, sessionID string, n int)
return msgs, nil return msgs, nil
} }
// newFakeConversations builds the single-session fixture every write-path
// case starts from: session "s1", served by agent "winter", with the given
// message tail.
func newFakeConversations(msgs ...curator.ConversationMessage) *fakeConversations {
return &fakeConversations{
sessions: map[string]time.Time{"s1": time.Unix(1000, 0)},
agents: map[string]string{"s1": "winter"},
messages: map[string][]curator.ConversationMessage{"s1": msgs},
}
}
// fakeLLM returns a canned response per call, or the same response every // fakeLLM returns a canned response per call, or the same response every
// time if only one is configured. // time if only one is configured.
type fakeLLM struct { type fakeLLM struct {
@@ -56,8 +69,12 @@ func (f *fakeLLM) Chat(context.Context, curator.ChatRequest) (curator.ChatResult
return curator.ChatResult{Text: f.responses[i]}, nil return curator.ChatResult{Text: f.responses[i]}, nil
} }
// fakeEngine records what the curator asked it to create. domain/memory's
// own fake engine is unexported in its package's test files, so this one is
// minimal on purpose: the read half exists only to satisfy MemoryEngine.
type fakeEngine struct { type fakeEngine struct {
writes []domainmemory.Observation created []domainmemory.NewRecord
createErr error
} }
func (f *fakeEngine) Name() string { return "builtin" } func (f *fakeEngine) Name() string { return "builtin" }
@@ -70,17 +87,45 @@ func (f *fakeEngine) Health(context.Context) domainmemory.Health {
} }
func (f *fakeEngine) Stop(context.Context) error { return nil } func (f *fakeEngine) Stop(context.Context) error { return nil }
func (f *fakeEngine) Write(_ context.Context, obs domainmemory.Observation) (domainmemory.Record, error) { func (f *fakeEngine) Create(_ context.Context, in domainmemory.NewRecord) (domainmemory.Record, error) {
f.writes = append(f.writes, obs) if f.createErr != nil {
return domainmemory.Record{ID: "id", Identity: obs.Identity, Content: obs.Content}, nil return domainmemory.Record{}, f.createErr
}
f.created = append(f.created, in)
return domainmemory.Record{
ID: domainmemory.RecordID(fmt.Sprintf("r%d", len(f.created))),
Scope: in.Scope,
Kind: in.Kind,
Content: in.Content,
Author: in.Author,
OriginUser: in.OriginUser,
Source: in.Source,
}, nil
}
func (f *fakeEngine) Get(context.Context, domainmemory.Scope, domainmemory.RecordID) (domainmemory.Record, error) {
return domainmemory.Record{}, domainmemory.ErrNotFound
} }
func (f *fakeEngine) Query(context.Context, domainmemory.Query) ([]domainmemory.Record, error) { func (f *fakeEngine) Query(context.Context, domainmemory.Query) ([]domainmemory.Record, error) {
return nil, nil return nil, nil
} }
func (f *fakeEngine) List(context.Context, string) ([]domainmemory.Record, error) { return nil, nil } func (f *fakeEngine) List(context.Context, domainmemory.Scope) ([]domainmemory.Record, error) {
func (f *fakeEngine) Forget(context.Context, string) error { return nil } return nil, nil
}
func (f *fakeEngine) Update(context.Context, domainmemory.RecordUpdate) (domainmemory.Record, error) {
return domainmemory.Record{}, domainmemory.ErrNotFound
}
func (f *fakeEngine) Forget(context.Context, domainmemory.Scope, domainmemory.RecordID) error {
return nil
}
func (f *fakeEngine) Revisions(context.Context, domainmemory.Scope, domainmemory.RecordID) ([]domainmemory.Revision, error) {
return nil, nil
}
type fakeEngineSource struct{ engine *fakeEngine } type fakeEngineSource struct{ engine *fakeEngine }
@@ -140,12 +185,7 @@ func TestCheckReportsTrueWithARecentSession(t *testing.T) {
func TestPassExtractsAndWritesObservations(t *testing.T) { func TestPassExtractsAndWritesObservations(t *testing.T) {
t.Parallel() t.Parallel()
conv := &fakeConversations{ conv := newFakeConversations(curator.ConversationMessage{Role: "user", SenderID: "u-42", Content: "I prefer tabs over spaces"})
sessions: map[string]time.Time{"s1": time.Unix(1000, 0)},
messages: map[string][]curator.ConversationMessage{
"s1": {{Role: "user", Content: "I prefer tabs over spaces"}},
},
}
llm := &fakeLLM{responses: []string{`["prefers tabs over spaces"]`}} llm := &fakeLLM{responses: []string{`["prefers tabs over spaces"]`}}
engine := &fakeEngine{} engine := &fakeEngine{}
c := newTestCurator(t, conv, llm, engine) c := newTestCurator(t, conv, llm, engine)
@@ -160,24 +200,192 @@ func TestPassExtractsAndWritesObservations(t *testing.T) {
if want := time.Unix(1000, 0); !res.Actions[0].At.Equal(want) { if want := time.Unix(1000, 0); !res.Actions[0].At.Equal(want) {
t.Errorf("Actions[0].At = %v, want %v (from the curator's clock, not zero)", res.Actions[0].At, want) t.Errorf("Actions[0].At = %v, want %v (from the curator's clock, not zero)", res.Actions[0].At, want)
} }
if len(engine.writes) != 1 { if len(engine.created) != 1 {
t.Fatalf("writes = %#v, want exactly 1", engine.writes) t.Fatalf("created = %#v, want exactly 1", engine.created)
} }
got := engine.writes[0] got := engine.created[0]
if got.Identity != "s1" { // The session id is not a scope: what the curator extracted is a fact
t.Errorf("Identity = %q, want the session id, not a user/cross-session identity", got.Identity) // about one person as seen by one agent, and agent-user is the only
// scope that says so.
if want := (domainmemory.Scope{Kind: domainmemory.ScopeAgentUser, Agent: "winter", User: "u-42"}); got.Scope != want {
t.Errorf("Scope = %v, want %v", got.Scope, want)
} }
if got.Content != "prefers tabs over spaces" { if got.Content != "prefers tabs over spaces" {
t.Errorf("Content = %q, want the extracted fact verbatim", got.Content) t.Errorf("Content = %q, want the extracted fact verbatim", got.Content)
} }
} }
// TestPassWritesExtractedFactsToAgentUserScope pins the address and
// provenance of every curator write: a later chat turn reads back
// agent-user memory for its own agent and user, and only a record written
// there is visible to it.
func TestPassWritesExtractedFactsToAgentUserScope(t *testing.T) {
t.Parallel()
tests := []struct {
name string
messages []curator.ConversationMessage
response string
wantContents []string
}{
{
name: "one user sender",
messages: []curator.ConversationMessage{{Role: "user", SenderID: "u-42", Content: "I prefer tabs"}},
response: `["prefers tabs over spaces"]`,
wantContents: []string{"prefers tabs over spaces"},
},
{
name: "the same sender across several messages is still one participant",
messages: []curator.ConversationMessage{
{Role: "user", SenderID: "u-42", Content: "I prefer tabs"},
{Role: "assistant", Content: "noted"},
{Role: "user", SenderID: "u-42", Content: "and vim bindings"},
},
response: `["prefers tabs over spaces"]`,
wantContents: []string{"prefers tabs over spaces"},
},
{
name: "an assistant sender beside one identified user sender is still one participant",
messages: []curator.ConversationMessage{
{Role: "user", SenderID: "u-42", Content: "I prefer tabs"},
{Role: "assistant", SenderID: "winter", Content: "noted"},
},
response: `["prefers tabs over spaces"]`,
wantContents: []string{"prefers tabs over spaces"},
},
{
name: "one write per extracted fact",
messages: []curator.ConversationMessage{{Role: "user", SenderID: "u-42", Content: "lots of facts"}},
response: `["fact a","fact b","fact c"]`,
wantContents: []string{"fact a", "fact b", "fact c"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
engine := &fakeEngine{}
c := newTestCurator(t, newFakeConversations(tt.messages...), &fakeLLM{responses: []string{tt.response}}, engine)
res, err := c.Pass(context.Background(), curator.PassInput{})
if err != nil {
t.Fatalf("Pass() = %v, want nil", err)
}
wantScope := domainmemory.Scope{Kind: domainmemory.ScopeAgentUser, Agent: "winter", User: "u-42"}
if len(engine.created) != len(tt.wantContents) {
t.Fatalf("created = %#v, want %d record(s)", engine.created, len(tt.wantContents))
}
for i, want := range tt.wantContents {
got := engine.created[i]
if got.Scope != wantScope {
t.Errorf("created[%d].Scope = %v, want %v", i, got.Scope, wantScope)
}
if got.Kind != "note" {
t.Errorf("created[%d].Kind = %q, want %q", i, got.Kind, "note")
}
if got.Author != Name {
t.Errorf("created[%d].Author = %q, want the producer's own name %q", i, got.Author, Name)
}
if got.OriginUser != domainmemory.IdentityID("u-42") {
t.Errorf("created[%d].OriginUser = %q, want the participant %q", i, got.OriginUser, "u-42")
}
if got.Source != "s1" {
t.Errorf("created[%d].Source = %q, want the session id %q", i, got.Source, "s1")
}
if got.Content != want {
t.Errorf("created[%d].Content = %q, want %q", i, got.Content, want)
}
}
if len(res.Actions) != 1 || res.Actions[0].Type != ActionExtracted {
t.Errorf("Actions = %#v, want one %s", res.Actions, ActionExtracted)
}
})
}
}
// TestPassSkipsSessionsWithoutASingleParticipant pins the fail-closed rule.
// Every case's model response offers a fact, so a write here could only come
// from a guessed participant.
func TestPassSkipsSessionsWithoutASingleParticipant(t *testing.T) {
t.Parallel()
tests := []struct {
name string
messages []curator.ConversationMessage
wantReason string
}{
{
name: "zero senders: a dashboard session carries no per-person identity",
messages: []curator.ConversationMessage{{Role: "user", Content: "hello"}},
wantReason: "no single participant: 0 distinct sender(s), 1 unidentified user message(s)",
},
{
name: "an identified sender beside an unidentified user message: the excerpt's provenance is unknown",
messages: []curator.ConversationMessage{
{Role: "user", SenderID: "u-42", Content: "I prefer tabs"},
{Role: "user", Content: "a message whose channel carries no sender id"},
},
wantReason: "no single participant: 1 distinct sender(s), 1 unidentified user message(s)",
},
{
name: "several distinct senders: a group chat",
messages: []curator.ConversationMessage{
{Role: "user", SenderID: "u-1", Content: "hi"},
{Role: "user", SenderID: "u-2", Content: "hello"},
},
wantReason: "no single participant: 2 distinct sender(s)",
},
{
name: "a sender present only on assistant messages does not count",
messages: []curator.ConversationMessage{{Role: "assistant", SenderID: "winter", Content: "hi"}},
wantReason: "no single participant: 0 distinct sender(s)",
},
{
name: "empty senders among the user messages do not create a participant",
messages: []curator.ConversationMessage{
{Role: "user", Content: "hello"},
{Role: "user", Content: "still nobody"},
},
wantReason: "no single participant: 0 distinct sender(s), 2 unidentified user message(s)",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
engine := &fakeEngine{}
c := newTestCurator(t, newFakeConversations(tt.messages...), &fakeLLM{responses: []string{`["a fact worth keeping"]`}}, engine)
res, err := c.Pass(context.Background(), curator.PassInput{})
if err != nil {
t.Fatalf("Pass() = %v, want nil: an unattributable session is skipped, not a pass failure", err)
}
if len(engine.created) != 0 {
t.Fatalf("created = %#v, want none: the session has no single participant to write for", engine.created)
}
if len(res.Actions) != 1 {
t.Fatalf("Actions = %#v, want exactly one skip", res.Actions)
}
got := res.Actions[0]
if got.Type != ActionSkipped {
t.Errorf("Actions[0].Type = %q, want %q", got.Type, ActionSkipped)
}
if got.Detail != "s1" {
t.Errorf("Actions[0].Detail = %q, want the session id %q", got.Detail, "s1")
}
if got.Reason != tt.wantReason {
t.Errorf("Actions[0].Reason = %q, want %q", got.Reason, tt.wantReason)
}
if want := time.Unix(1000, 0); !got.At.Equal(want) {
t.Errorf("Actions[0].At = %v, want %v (from the curator's clock)", got.At, want)
}
})
}
}
func TestPassTreatsUnparseableResponseAsNothingExtracted(t *testing.T) { func TestPassTreatsUnparseableResponseAsNothingExtracted(t *testing.T) {
t.Parallel() t.Parallel()
conv := &fakeConversations{ conv := newFakeConversations(curator.ConversationMessage{Role: "user", SenderID: "u-42", Content: "hello"})
sessions: map[string]time.Time{"s1": time.Unix(1000, 0)},
messages: map[string][]curator.ConversationMessage{"s1": {{Role: "user", Content: "hello"}}},
}
llm := &fakeLLM{responses: []string{"not json at all"}} llm := &fakeLLM{responses: []string{"not json at all"}}
engine := &fakeEngine{} engine := &fakeEngine{}
c := newTestCurator(t, conv, llm, engine) c := newTestCurator(t, conv, llm, engine)
@@ -189,17 +397,14 @@ func TestPassTreatsUnparseableResponseAsNothingExtracted(t *testing.T) {
if len(res.Actions) != 0 { if len(res.Actions) != 0 {
t.Fatalf("Actions = %#v, want none", res.Actions) t.Fatalf("Actions = %#v, want none", res.Actions)
} }
if len(engine.writes) != 0 { if len(engine.created) != 0 {
t.Fatalf("writes = %#v, want none", engine.writes) t.Fatalf("created = %#v, want none", engine.created)
} }
} }
func TestPassEmptyArrayExtractsNothing(t *testing.T) { func TestPassEmptyArrayExtractsNothing(t *testing.T) {
t.Parallel() t.Parallel()
conv := &fakeConversations{ conv := newFakeConversations(curator.ConversationMessage{Role: "user", SenderID: "u-42", Content: "hi"})
sessions: map[string]time.Time{"s1": time.Unix(1000, 0)},
messages: map[string][]curator.ConversationMessage{"s1": {{Role: "user", Content: "hi"}}},
}
llm := &fakeLLM{responses: []string{`[]`}} llm := &fakeLLM{responses: []string{`[]`}}
engine := &fakeEngine{} engine := &fakeEngine{}
c := newTestCurator(t, conv, llm, engine) c := newTestCurator(t, conv, llm, engine)
@@ -213,12 +418,27 @@ func TestPassEmptyArrayExtractsNothing(t *testing.T) {
} }
} }
func TestPassTreatsASessionWithNoMessagesAsNothingExtracted(t *testing.T) {
t.Parallel()
conv := newFakeConversations()
engine := &fakeEngine{}
c := newTestCurator(t, conv, &fakeLLM{responses: []string{`["a fact worth keeping"]`}}, engine)
res, err := c.Pass(context.Background(), curator.PassInput{})
if err != nil {
t.Fatalf("Pass() = %v, want nil", err)
}
if len(res.Actions) != 0 {
t.Fatalf("Actions = %#v, want none: a session with no messages yields nothing to extract", res.Actions)
}
if len(engine.created) != 0 {
t.Fatalf("created = %#v, want none", engine.created)
}
}
func TestPassCapsObservationsPerSession(t *testing.T) { func TestPassCapsObservationsPerSession(t *testing.T) {
t.Parallel() t.Parallel()
conv := &fakeConversations{ conv := newFakeConversations(curator.ConversationMessage{Role: "user", SenderID: "u-42", Content: "lots of facts"})
sessions: map[string]time.Time{"s1": time.Unix(1000, 0)},
messages: map[string][]curator.ConversationMessage{"s1": {{Role: "user", Content: "lots of facts"}}},
}
llm := &fakeLLM{responses: []string{`["a","b","c","d","e","f","g"]`}} llm := &fakeLLM{responses: []string{`["a","b","c","d","e","f","g"]`}}
engine := &fakeEngine{} engine := &fakeEngine{}
c := newTestCurator(t, conv, llm, engine) c := newTestCurator(t, conv, llm, engine)
@@ -226,8 +446,15 @@ func TestPassCapsObservationsPerSession(t *testing.T) {
if _, err := c.Pass(context.Background(), curator.PassInput{}); err != nil { if _, err := c.Pass(context.Background(), curator.PassInput{}); err != nil {
t.Fatalf("Pass() = %v, want nil", err) t.Fatalf("Pass() = %v, want nil", err)
} }
if len(engine.writes) != maxObservationsPerSession { if len(engine.created) != maxObservationsPerSession {
t.Fatalf("writes = %d, want capped at %d", len(engine.writes), maxObservationsPerSession) t.Fatalf("created = %d, want capped at %d", len(engine.created), maxObservationsPerSession)
}
// The cap truncates the tail, and keeps the order the model returned.
want := []string{"a", "b", "c", "d", "e"}
for i, content := range want {
if engine.created[i].Content != content {
t.Errorf("created[%d].Content = %q, want %q", i, engine.created[i].Content, content)
}
} }
} }
@@ -235,12 +462,16 @@ func TestPassReviewsMultipleSessionsIndependently(t *testing.T) {
t.Parallel() t.Parallel()
conv := &fakeConversations{ conv := &fakeConversations{
sessions: map[string]time.Time{"s1": time.Unix(1000, 0), "s2": time.Unix(1000, 0)}, sessions: map[string]time.Time{"s1": time.Unix(1000, 0), "s2": time.Unix(1000, 0)},
agents: map[string]string{"s1": "winter", "s2": "spring"},
messages: map[string][]curator.ConversationMessage{ messages: map[string][]curator.ConversationMessage{
"s1": {{Role: "user", Content: "fact for s1"}}, "s1": {{Role: "user", SenderID: "u-1", Content: "fact for s1"}},
"s2": {{Role: "user", Content: "fact for s2"}}, "s2": {{Role: "user", SenderID: "u-2", Content: "fact for s2"}},
}, },
} }
llm := &fakeLLM{responses: []string{`["fact one"]`, `["fact two"]`}} // One response for both sessions: the map iteration order above is not
// deterministic, so nothing here may depend on which session is asked
// first -- attribution is asserted through Source.
llm := &fakeLLM{responses: []string{`["a fact"]`}}
engine := &fakeEngine{} engine := &fakeEngine{}
c := newTestCurator(t, conv, llm, engine) c := newTestCurator(t, conv, llm, engine)
@@ -251,15 +482,23 @@ func TestPassReviewsMultipleSessionsIndependently(t *testing.T) {
if len(res.Actions) != 2 { if len(res.Actions) != 2 {
t.Fatalf("Actions = %#v, want 2 (one per session)", res.Actions) t.Fatalf("Actions = %#v, want 2 (one per session)", res.Actions)
} }
if len(engine.writes) != 2 { if len(engine.created) != 2 {
t.Fatalf("writes = %#v, want 2", engine.writes) t.Fatalf("created = %#v, want 2", engine.created)
} }
identities := map[string]bool{} scopes := map[string]domainmemory.Scope{}
for _, w := range engine.writes { for _, rec := range engine.created {
identities[w.Identity] = true scopes[rec.Source] = rec.Scope
} }
if !identities["s1"] || !identities["s2"] { // Each session's own agent addresses its scope, so a session summary
t.Errorf("writes scoped to identities %v, want both s1 and s2", identities) // whose agent id is dropped or swapped writes into another agent's
// memory.
for session, want := range map[string]domainmemory.Scope{
"s1": {Kind: domainmemory.ScopeAgentUser, Agent: "winter", User: "u-1"},
"s2": {Kind: domainmemory.ScopeAgentUser, Agent: "spring", User: "u-2"},
} {
if got := scopes[session]; got != want {
t.Errorf("scope of the write sourced from %s = %v, want %v", session, got, want)
}
} }
} }
@@ -340,17 +579,17 @@ func TestRegistersAndRunsThroughTheRegistry(t *testing.T) {
} }
} }
var errLLM = errors.New("model unavailable") var (
errLLM = errors.New("model unavailable")
errCreate = errors.New("engine refused the write")
)
func TestPassPropagatesALLMFailure(t *testing.T) { func TestPassPropagatesALLMFailure(t *testing.T) {
// Unlike an unparseable response, the model call itself failing (the // Unlike an unparseable response, the model call itself failing (the
// provider is down) is a real Pass error, not a "nothing to extract" // provider is down) is a real Pass error, not a "nothing to extract"
// outcome. // outcome.
t.Parallel() t.Parallel()
conv := &fakeConversations{ conv := newFakeConversations(curator.ConversationMessage{Role: "user", SenderID: "u-42", Content: "hi"})
sessions: map[string]time.Time{"s1": time.Unix(1000, 0)},
messages: map[string][]curator.ConversationMessage{"s1": {{Role: "user", Content: "hi"}}},
}
llm := &fakeLLM{err: errLLM} llm := &fakeLLM{err: errLLM}
c := newTestCurator(t, conv, llm, &fakeEngine{}) c := newTestCurator(t, conv, llm, &fakeEngine{})
@@ -358,3 +597,17 @@ func TestPassPropagatesALLMFailure(t *testing.T) {
t.Fatalf("Pass() error = %v, want it to wrap the LLM failure", err) t.Fatalf("Pass() error = %v, want it to wrap the LLM failure", err)
} }
} }
func TestPassPropagatesACreateFailure(t *testing.T) {
// The engine refusing a write is a real failure, not "nothing to
// extract": the pass must not report success for a fact it did not
// persist.
t.Parallel()
conv := newFakeConversations(curator.ConversationMessage{Role: "user", SenderID: "u-42", Content: "hi"})
engine := &fakeEngine{createErr: errCreate}
c := newTestCurator(t, conv, &fakeLLM{responses: []string{`["a fact"]`}}, engine)
if _, err := c.Pass(context.Background(), curator.PassInput{}); !errors.Is(err, errCreate) {
t.Fatalf("Pass() error = %v, want it to wrap the engine's create failure", err)
}
}
+26 -117
View File
@@ -1,123 +1,32 @@
package memory package memory
import "regexp" import domainmemory "github.com/samcharles93/archie-core/internal/domain/memory"
// ThreatLevel indicates the severity of a threat detected by a Scanner. // The content scanner moved into the memory engine family
type ThreatLevel int // (internal/domain/memory/scanner.go, docs/prds/memory-engine-unification.md
// §7), where the engine that persists memory applies it. This package's
// legacy Manager still needs the same types until it is deleted in slice 5 of
// that PRD, so they are re-exported here rather than duplicated: the
// behaviour, the patterns and the tests stay exactly where the decision put
// them.
//
// Delete this file with internal/memory.
type (
// ThreatLevel is domainmemory.ThreatLevel.
ThreatLevel = domainmemory.ThreatLevel
// ThreatResult is domainmemory.ThreatResult.
ThreatResult = domainmemory.ThreatResult
// Scanner is domainmemory.Scanner.
Scanner = domainmemory.Scanner
// DefaultScanner is domainmemory.DefaultScanner.
DefaultScanner = domainmemory.DefaultScanner
)
const ( const (
// ThreatNone means no suspicious patterns were found. // ThreatNone is domainmemory.ThreatNone.
ThreatNone ThreatLevel = iota ThreatNone = domainmemory.ThreatNone
// ThreatWarn is domainmemory.ThreatWarn.
// ThreatWarn means a potentially problematic pattern was found, but the ThreatWarn = domainmemory.ThreatWarn
// write should be allowed with a warning logged. // ThreatBlock is domainmemory.ThreatBlock.
ThreatWarn ThreatBlock = domainmemory.ThreatBlock
// ThreatBlock means a dangerous pattern was found and the write should
// be rejected.
ThreatBlock
) )
// ThreatResult describes a single threat detected during content scanning.
type ThreatResult struct {
// Level indicates the severity: none, warn, or block.
Level ThreatLevel
// Pattern is the regexp that matched (for audit logging).
Pattern string
// Message is a human-readable description of the threat.
Message string
}
// Scanner inspects memory content for dangerous or sensitive patterns before
// the content is persisted. Implementations are pluggable; the zero-value
// DefaultScanner is a reasonable default for production use.
type Scanner interface {
// ScanContent examines content and returns a ThreatResult. A result
// with Level == ThreatNone means the content is clean.
ScanContent(content string) ThreatResult
}
// DefaultScanner is the built-in Scanner implementation. It checks for
// prompt-injection patterns (block-level) and sensitive-data leakage
// (warn-level). The zero value is usable.
type DefaultScanner struct{}
// Compile-time check: DefaultScanner implements Scanner.
var _ Scanner = (*DefaultScanner)(nil)
// ── Patterns ────────────────────────────────────────────────────────────
//
// These patterns are compiled once at init time. They are intentionally
// broad to catch common variants without being so broad that they
// produce false positives on normal memory content.
var (
promptInjectionPatterns = []*regexp.Regexp{
// Override / ignore previous instructions
regexp.MustCompile(`(?i)(ignore|forget|override|disregard)\s+(all\s+)?(the\s+)?(previous|above|prior|earlier)\s+(instructions?|directions?|commands?|prompts?)`),
// Role-change / identity attacks
regexp.MustCompile(`(?i)(you\s+(are|now)\s+|act\s+as\s+|pretend\s+(to\s+be|you\s+are)\s+|roleplay\s+as\s+)(now\s+)?(an?\s+)?(different|new|another|unrestricted|uncensored)`),
// Delimiter injection (common in adversarial prompts)
regexp.MustCompile(`(?i)<\|im_start\|>`),
regexp.MustCompile(`(?i)<\|im_end\|>`),
// System prompt extraction attempts
regexp.MustCompile(`(?i)(print|output|repeat|display|show|tell\s+me)\s+(your\s+)?(system\s+)?(prompt|instructions?|directions?)\s*(exactly|verbatim)?`),
}
sensitiveDataPatterns = []*regexp.Regexp{
// API keys and secrets with common key=value patterns
regexp.MustCompile(`(?i)(api[_\s-]?key|apikey|api[_\s-]?secret)\s*[:=]\s*['"\x60]?[\w-]{8,}['"\x60]?`),
// Passwords in key=value form
regexp.MustCompile(`(?i)(password|passwd|pwd)\s*[:=]\s*['"\x60]?\S{4,}['"\x60]?`),
// Generic secrets and tokens
regexp.MustCompile(`(?i)(secret|token|credential|authorization)\s*[:=]\s*['"\x60]?[\w-]{8,}['"\x60]?`),
// Private key blocks (PEM format)
regexp.MustCompile(`(?i)-----BEGIN\s+(RSA\s+|EC\s+|DSA\s+|OPENSSH\s+)?PRIVATE\s+KEY-----`),
// JWT tokens (base64-encoded header.payload.signature pattern)
regexp.MustCompile(`eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}`),
}
)
// ScanContent examines content against the prompt-injection and
// sensitive-data patterns. Prompt-injection matches produce ThreatBlock;
// sensitive-data matches produce ThreatWarn. The first match wins.
func (s *DefaultScanner) ScanContent(content string) ThreatResult {
for _, p := range promptInjectionPatterns {
if m := p.FindString(content); m != "" {
return ThreatResult{
Level: ThreatBlock,
Pattern: p.String(),
Message: "content matches prompt injection pattern: " + truncateMatch(m),
}
}
}
for _, p := range sensitiveDataPatterns {
if m := p.FindString(content); m != "" {
return ThreatResult{
Level: ThreatWarn,
Pattern: p.String(),
Message: "content may contain sensitive data: " + truncateMatch(m),
}
}
}
return ThreatResult{Level: ThreatNone}
}
// truncateMatch limits the matched string for log/error messages so that
// a long match (e.g. an entire private key) is not echoed in full.
func truncateMatch(s string) string {
const maxMatchLen = 40
if len(s) <= maxMatchLen {
return s
}
return s[:maxMatchLen] + "..."
}