session-memory curator: summarise a section's oldest notes before it reaches the size cap #1669

Open
opened 2026-10-27 13:41:19 +00:00 by sam · 1 comment
Owner

The builtin memory engine now rotates a full section's oldest records to HISTORY.md so writes never fail, and a full HISTORY.md drops its oldest state for good. Rotation keeps recording but loses old observations from what chat turns read back. The curator should compress them instead: when a section nears the cap, summarise its oldest notes into one record with a model call and retire the originals to history. Rotation stays as the fallback when the model is unavailable or the pass fails.

Design

Also mark rotated history entries distinctly from Forget deletions (a rotated flag on the history marker) so history shows why a record left.

Acceptance criteria

A section approaching the cap gets its oldest notes replaced by a summary record during a curator pass, the originals appear in history, and a pass with no model available still succeeds via rotation.


Imported from beads archie-core-y9mx0 · GitHub #1910 · type feature · priority P3 · created 2026-10-25

The builtin memory engine now rotates a full section's oldest records to HISTORY.md so writes never fail, and a full HISTORY.md drops its oldest state for good. Rotation keeps recording but loses old observations from what chat turns read back. The curator should compress them instead: when a section nears the cap, summarise its oldest notes into one record with a model call and retire the originals to history. Rotation stays as the fallback when the model is unavailable or the pass fails. ### Design Also mark rotated history entries distinctly from Forget deletions (a rotated flag on the history marker) so history shows why a record left. ### Acceptance criteria A section approaching the cap gets its oldest notes replaced by a summary record during a curator pass, the originals appear in history, and a pass with no model available still succeeds via rotation. --- _Imported from beads `archie-core-y9mx0` · GitHub #1910 · type feature · priority P3 · created 2026-10-25_
sam added the lowfeature labels 2026-10-27 13:41:19 +00:00
sam self-assigned this 2026-10-27 13:41:19 +00:00
Author
Owner

archie's PRD is ready -- awaiting your go/no-go.

PRD — session-memory curator: compress a scope's oldest notes instead of only rotating them

Issue sam/archie#1669 (archie-core-y9mx0, P3, feature). Recommendation: green-light as specified below, but not before the 1 December production-readiness work — this is a quality improvement, not a fix for lost work (rotation already bounds loss).

1. Problem

The builtin memory engine already degrades gracefully, in a way that loses information from the turn's point of view:

  • BuiltinEngine.Create retries builtin.Store.Add on builtin.ErrTooLarge and calls retireOldest, which appends the oldest block of the section being written to that scope's HISTORY.md and removes it live (internal/infrastructure/memory/builtin_engine.go; test TestBuiltinEngineCreateRotatesFullSection).
  • appendHistory (history.go) drops the oldest history state once HISTORY.md reaches historyMaxFileBytes = 4 MB, so history itself is a bounded ring.
  • Nothing reads HISTORY.md. engine.Revisions is called from exactly one place, builtin_engine_test.go:34; the gateway's MemoryStore / MemoryWriteStore narrow the engine contract and expose nothing from history. A rotated record is therefore invisible to every part of the product.

With concrete numbers, the true cost of rotation is: a chat turn reads at most memoryRecordsPerScope = 20 records per scope (internal/gateway/turn_memory.go, one Query per scope with Limit: 20, under an 8192-byte block cap). A scope at the default defaultMaxFileBytes = 100 KB holds ~380 note blocks (~210 bytes of marker JSON + short content), i.e. ~20 read-windows. Rotating one record therefore changes nothing a turn sees — until the live scope drops below 20 records, at which point old material has silently fallen out of the read window forever. Rotation keeps the write path alive and keeps provenance on disk, but it does not keep old observations in front of the model.

The feature: when a scope approaches its size bound, have the session-memory curator replace that scope's oldest notes with one summary record, so a compressed stand-in for the retired material reappears inside the 20-record read window, and retire the originals to HISTORY.md marked as rotated rather than forgotten. Rotation stays the fallback when the model is unavailable or the pass fails.

2. What the issue assumes that is not true — and how the PRD corrects it

  1. "a section's oldest notes" — the cap is per scope document, not per section. builtin.Store.Add rejects when len(renderSections(...)) > s.maxBytes over the whole OBSERVATIONS.md; sections are just markdown groupings inside one file. retireOldest happens to pick the oldest block whose section matches the section being written, but the budget it is defending is the file's. PRD language: "the scope's live document".
  2. Occupancy is not visible outside the engine. BuiltinEngine.maxFileBytes is private; builtin.Store has Size()/MaxBytes() but domain/memory.Store exposes neither and Manifest carries only RequiresNetwork. A curator cannot tell whether a scope is near its cap, so the feature needs a new read/maintenance surface (§4).
  3. No scope enumeration. e.stores is keyed by scope.Key() and nothing lists scopes. sessioncurator.Pass derives scopes only from Conversations.RecentSessions, so the curator can only ever touch agent-user scopes of recently active sessions. Cold scopes, agent scopes and global scope are out of reach (non-goal §7).
  4. retireOldest marks rotation as a deletion today. It calls appendHistory(..., true), setting markerData.Deleted — the same flag Forget sets, with no way to tell them apart. The issue's request (a rotated flag) is real and also fixes a mislabel: a rotated record was not deleted by anyone.
  5. AC3 ("a pass with no model available still succeeds via rotation") cannot happen as wired. archiegateway.setupCurators returns false and registers no curators when chatModels.ModelFor fails ("curators disabled: chat model unavailable; configure a chat model and restart the gateway"), and a provider failure at pass time propagates out of reviewOne and fails the whole pass. Either AC3 means "a pass where the compaction model call fails", or curator startup has to change for the whole family (skill curator and definition engine included). §6 adopts the first reading and §9 flags it for confirmation.
  6. The "cheaper, model-free coalescing" alternative does not work. Merging N note blocks into one "earlier observations" record changes record count, not bytes — the document is exactly as full afterwards, so it never makes room and rotation still fires at the cap. Only lossy compression reduces occupancy. The model call is load-bearing, not an optional upgrade. (The one thing deterministic merging would fix is the 20-record window; a summary fixes both.)

3. Goals

  • A scope above a size threshold gets its oldest records replaced by one smaller summary record during a session-memory curator pass.
  • The replaced originals are retained in HISTORY.md and are distinguishable from Forget deletions there.
  • A failed or unavailable summary degrades to today's behaviour: no write, no failed pass, rotation still bounds the document.
  • No new configuration surface, no new service, no new process.

4. Proposed solution

4.1 Domain contract (internal/domain/memory/contract.go) — one new operation

// Summarizer compresses a scope's oldest records into one replacement
// record's content. The curator supplies the model call; the engine owns
// where and when it is applied.
type Summarizer func(ctx context.Context, contents []string) (string, error)

// Compact replaces q's oldest records with one smaller record when the
// scope's live document is near its bound. Below the threshold it is a
// no-op with a zero result and no call to summarise.
Compact(ctx context.Context, scope Scope, summarise Summarizer) (CompactionResult, error)

type CompactionResult struct {
    OccupancyBefore, OccupancyAfter int  // bytes of the live document
    SummaryID                       RecordID
    Retired                         []RecordID
}

Added to Store (not an optional interface): BuiltinEngine is the only implementer — verified, no other type implements Manifest() domainmemory.Manifest and no fake implements the full domainmemory.Store (the gateway's narrowed interfaces are unaffected). The engine keeps the format knowledge, the atomic ordering and the threshold; the curator contributes only the model call. It also means the curator needs no occupancy query: below the threshold Compact returns immediately without calling summarise.

Also on the contract: Revision gains Rotated bool beside Deleted bool.

4.2 Builtin engine (internal/infrastructure/memory/builtin_engine.go)

Algorithm for Compact:

  1. Resolve the scope's stores read-only; no directory → zero result.
  2. stores.live.Reload() first, so a write made by another process holding the file is visible.
  3. Occupancy = len(stores.live.Render()); if occupancy < compactThresholdRatio (0.8) * maxBytes → return zero result, nil.
  4. Candidates = live blocks sorted by UpdatedAt ascending, taken inline-document order for ties, up to compactMaxCandidates (32) or compactBudgetBytes (8 KB of content), whichever comes first. Every kind is eligible, including a previous summary (gradual forgetting is intended; the ≤50%-smaller guard bounds the damage). Newest records are never candidates.
  5. Call summarise(ctx, contents). Error → return it without writing anything.
  6. Size guard: if len(summary) >= 0.6 * (bytes of candidates replaced), write nothing and return an error ("summary did not make room"). This is what stops a "summary" that is not a win from retiring records for nothing.
  7. Write order, chosen so a crash never leaves a fact counted twice:
    a. appendHistory(..., rotated=true) for each candidate;
    b. stores.live.Remove(...) each candidate — a missing block is success, not an error (another process may have rotated it);
    c. stores.live.Add("summary", renderBlock(...)) the summary last, so the file has room by construction and Add cannot re-enter rotation.
    A crash between (b) and (c) leaves the originals retained in history with no summary — exactly today's guarantee, not a new loss.
  8. Summary record provenance: Kind: "summary", Author: "session-memory curator", Source: "compaction of N records", Revision: 1. It lands in a ## summary section (valid under builtin.ValidateSectionName); it is the newest record, so it sits at the top of the 20-record read window.

Concurrency: BuiltinEngine.mu guards only the store map and builtin.Store locks per operation, so a compaction is not atomic against a concurrent chat-turn Create. The Reload + missing-block tolerance above is the mitigation; a concurrent write in the middle can cost one duplicate summary line, never a lost record.

4.3 History marking (history.go, marker.go)

  • markerData gains Rotated bool \json:"rotated,omitempty"`; revisionsOfmaps it intoRevision.Rotated`.
  • appendHistory takes an explicit reason instead of the lone deleted bool — e.g. (deleted, rotated bool):
    • Update → (false, false)
    • Forget → (true, false)
    • retireOldest → (false, true) ← behaviour change: rotation is no longer labelled a deletion
    • Compact → (false, true)
  • No backfill: pre-existing rotated entries keep deleted: true because nothing recorded which reason applied. Harmless — no reader today — and stated so the reviewer is not surprised by mixed markers.

4.4 Curator (internal/infrastructure/sessioncurator/sessioncurator.go)

  • Pass collects map[domainmemory.Scope]struct{} of every scope it considered while iterating sessions (built before the model call in reviewOne, so a skipped or empty session still contributes its scope).
  • After the extraction loop, for each scope (bounded to maxCompactionsPerPass = 3, deterministic order by scope key):
    engine.Compact(ctx, scope, c.summarise).
  • c.summarise is one model call with a new compactPrompt ("merge these observations into one record of at most X words, dropping anything superseded, dated or one-off; keep names and preferences verbatim"), reusing c.host.LLM / c.host.Model exactly as askModel does.
  • Error isolation: any error from Compact (model failure, no room made, engine error) becomes an action, not a Pass error:
    • ActionCompacted = "memory.compacted", detail "scope <kind>: <n> record(s) → 1 summary, <before> → <after> bytes";
    • ActionCompactionSkipped = "memory.compaction_skipped", detail = the error, reason = why.
  • Extraction failures keep their current semantics (a real provider failure still fails the pass); the change is scoped to compaction. This keeps a compaction problem from ever costing a pass's extracted observations.
  • Budget: a pass makes at most 3 compaction attempts, and zero model calls when no scope is above the threshold, because the engine short-circuits. That matters because passes share a single 10-minute slot process-wide (defaultPassTimeout, MaxConcurrentPasses default 1).

4.5 The reader for the rotated flag

Storing the flag alone would be "a field with no consumer" — AGENTS.md's complaint. Two readers, no new UI:

  1. Revision.Rotated through the existing Revisions contract method (engine-level verification, and the seam any future history viewer uses).
  2. User-visible: the memory.compacted / memory.compaction_skipped curator actions appear in CuratorHealth.recent_actions (proto/gateway/v1/chat.proto) and render on the existing dashboard page /settings/curators (ui/src/curators/CuratorCard.vue, ring buffer of 20 in domain/curator/activity.go). That is the human-facing "why did these records leave".

A HISTORY.md / memory browser is a non-goal (§7) and a separate issue; there is no memory page in the dashboard at all today.

4.6 No new configuration

Deliberately no max_file_bytes or compact_threshold field. MemoryConfig carries only Engine, config.example.toml has no [memory] block, and controlplanestore has no memory settings kind — a config field would need a new control-plane kind and dashboard form to satisfy "a setting must work end to end". Instead the engine reads its own cap (Store.MaxBytes()) and applies named constants (compactThresholdRatio = 0.8, compactMaxCandidates = 32, compactBudgetBytes = 8 KB, maxCompactionsPerPass = 3). If tuning becomes necessary, that is a separate memory-settings control-plane issue.

5. Files and components affected

File Change Est. lines
internal/domain/memory/contract.go Summarizer, CompactionResult, Store.Compact doc+signature, Revision.Rotated ~45
internal/infrastructure/memory/builtin_engine.go Compact + candidate selection + constants ~110
internal/infrastructure/memory/history.go appendHistory(deleted, rotated), revisionsOf maps Rotated ~12
internal/infrastructure/memory/marker.go Rotated field ~4
internal/infrastructure/memory/builtin_engine.go (retireOldest) pass rotated instead of deleted ~2
internal/infrastructure/sessioncurator/sessioncurator.go scope collection, summarise, compactPrompt, 2 action types, error isolation ~90
internal/infrastructure/memory/builtin_engine_test.go + a sessioncurator test new cases (table-driven) ~120
Dashboard / proto / config / gateway wiring none 0

6. Acceptance criteria

Test seam changes here (the memory engine contract and provenance semantics), so tests are warranted per AGENTS.md.

  1. Compaction happens. Given a scope whose OBSERVATIONS.md is at ≥80% of its cap, one session-memory curator pass over that scope replaces its oldest records with exactly one live summary record, and the scope's live byte count falls. Verified by a table-driven engine test with a stub summariser and a fixed maxBytes (following TestBuiltinEngineCreateRotatesFullSection's NewBuiltinEngine(t.TempDir(), 4096)).
  2. Originals are retained and marked. Each replaced record returns Revisions entries with Rotated == true and Deleted == false, content unchanged; a record removed via Forget still returns Deleted == true, Rotated == false; an Update produces neither flag.
  3. Rotation is untouched as the fallback. TestBuiltinEngineCreateRotatesFullSection still passes unfiltered, and now also asserts Rotated == true on the retained record.
  4. Degradation is fail-soft. With a summariser that returns an error, and separately one that returns a summary larger than 60% of its input: Compact returns that error, OBSERVATIONS.md and HISTORY.md are byte-identical to before the call, the curator pass returns a memory.compaction_skipped action and no error, and further Create calls still succeed (rotation still fires at the cap).
  5. Below the threshold is free. Compact on a scope under 80% returns a zero result, writes nothing, does not call the summariser, and the pass makes no extra model call — assert the stub summariser's call count is 0.
  6. No new configuration exists. config.example.toml and MemoryConfig are unchanged; a reviewer can confirm git diff --stat touches no file under internal/config, proto/ or ui/.

7. Explicit non-goals

  • No history/memory viewer UI; no HISTORY.md reader beyond Revisions. Follow-up issue.
  • No scope enumeration (Scopes() on the engine). Scopes with no recently active session are not compacted; rotation remains their bound. Follow-up if cold scopes matter.
  • No change to curator startup without a chat model (§9, Q1). The skill curator and definition engine would be affected, so it is a separate issue.
  • No change to the 100 KB live cap, the 4 MB history cap, memoryRecordsPerScope (20) or the 8192-byte prompt block cap; no ranked retrieval.
  • No id linkage from a history entry back to the summary that replaced it (replaced_by in the marker). ~3 extra lines if the reviewer wants it; recommend deferring, since nothing reads it.
  • No record pinning / "never summarise this"; no per-scope policy.
  • No migration or rewrite of existing OBSERVATIONS.md/HISTORY.md files.
  • Not a fix for a lost-work bug: rotation already bounds loss, hence P3 and no place in the 1 December push.

8. Risks

  • Summary quality is a prompt problem with a size guard, not a correctness guarantee. A bad summary can drop a fact that a turn would otherwise have read. Bounded by the 60%-smaller guard and by history retention; accepted for a solo-instance product. The alternative (rotation) is strictly worse for the model's view.
  • Summary-of-summary degradation across cycles. Accepted; each cycle must halve to be accepted, so it is slow.
  • Non-atomic against concurrent writers (§4.2). Mitigated by Reload, missing-block tolerance and summary-last ordering; worst case is a duplicated line.
  • Extra model spend: at most 3 calls per hourly pass, zero when no scope is near its cap. Notes accumulate ~5 facts × ~250 bytes per session, so one 32-candidate compaction (~8 KB → ~3 KB) outweighs a typical hour of growth per scope.

9. Two decisions I need confirmed (recommendations given)

  1. AC3 as written is unachievable. "A pass with no model available still succeeds via rotation" — there is no pass without a model: setupCurators registers no curators at all when chatModels.ModelFor fails. I recommend restating it as AC4 above ("a pass whose compaction model call fails still succeeds, and rotation still bounds the document"). If you want the literal reading, the change is in archiegateway.setupCurators plus a nil-LLM guard in the session-memory curator, and it also changes skill-curator and definition-engine startup — a separate issue, and I would not bundle it here.
  2. Is the model call required, or is any size-reducing replacement acceptable? I recommend required (it is the only thing that reduces bytes, §2.6), with the guard in §4.2/6 that a non-shrinking result is rejected rather than applied.

Estimated diff size

~265 lines of production code + ~120 lines of tests ≈ 385 lines across 6 files in 3 packages (domain/memory, infrastructure/memory, infrastructure/sessioncurator), zero changes to proto, config, dashboard or the composition root. Well past AGENTS.md's ~100-line "stop and assess" mark, which is why this is a PRD and not a patch.

Verified vs not verified

Verified by reading the tree: every code reference, constant and call site above; the AC3 contradiction; the absence of a control-plane memory settings kind; Revisions having only one (test) caller; BuiltinEngine being the sole engine implementation; no fake implementing the full domain/memory.Store. Not verified: no code was run and no test executed (task check not run); the ~260-byte note-block estimate and the 0.8/32/8 KB tuning values are arithmetic from the marker format, not measurements; whether the operator wants Rotated surfaced anywhere beyond the curator actions is their call (§9 Q1 is the only blocking one).

Decide in chat or on the dashboard.

**archie's PRD is ready -- awaiting your go/no-go.** # PRD — session-memory curator: compress a scope's oldest notes instead of only rotating them Issue sam/archie#1669 (`archie-core-y9mx0`, P3, feature). Recommendation: **green-light as specified below**, but not before the 1 December production-readiness work — this is a quality improvement, not a fix for lost work (rotation already bounds loss). ## 1. Problem The builtin memory engine already degrades gracefully, in a way that loses information from the turn's point of view: - `BuiltinEngine.Create` retries `builtin.Store.Add` on `builtin.ErrTooLarge` and calls `retireOldest`, which appends the oldest block **of the section being written** to that scope's `HISTORY.md` and removes it live (`internal/infrastructure/memory/builtin_engine.go`; test `TestBuiltinEngineCreateRotatesFullSection`). - `appendHistory` (`history.go`) drops the oldest history state once `HISTORY.md` reaches `historyMaxFileBytes` = 4 MB, so history itself is a bounded ring. - Nothing reads `HISTORY.md`. `engine.Revisions` is called from exactly one place, `builtin_engine_test.go:34`; the gateway's `MemoryStore` / `MemoryWriteStore` narrow the engine contract and expose nothing from history. A rotated record is therefore invisible to every part of the product. With concrete numbers, the true cost of rotation is: a chat turn reads at most `memoryRecordsPerScope` = 20 records **per scope** (`internal/gateway/turn_memory.go`, one `Query` per scope with `Limit: 20`, under an 8192-byte block cap). A scope at the default `defaultMaxFileBytes` = 100 KB holds ~380 note blocks (~210 bytes of marker JSON + short content), i.e. ~20 read-windows. Rotating one record therefore changes nothing a turn sees — until the live scope drops below 20 records, at which point old material has silently fallen out of the read window forever. Rotation keeps the write path alive and keeps provenance on disk, but it does not keep old observations in front of the model. The feature: when a scope approaches its size bound, have the session-memory curator replace that scope's oldest notes with **one summary record**, so a compressed stand-in for the retired material reappears inside the 20-record read window, and retire the originals to `HISTORY.md` marked as rotated rather than forgotten. Rotation stays the fallback when the model is unavailable or the pass fails. ## 2. What the issue assumes that is not true — and how the PRD corrects it 1. **"a section's oldest notes"** — the cap is **per scope document, not per section**. `builtin.Store.Add` rejects when `len(renderSections(...)) > s.maxBytes` over the whole `OBSERVATIONS.md`; sections are just markdown groupings inside one file. `retireOldest` happens to pick the oldest block whose `section` matches the section being written, but the budget it is defending is the file's. **PRD language: "the scope's live document".** 2. **Occupancy is not visible outside the engine.** `BuiltinEngine.maxFileBytes` is private; `builtin.Store` has `Size()`/`MaxBytes()` but `domain/memory.Store` exposes neither and `Manifest` carries only `RequiresNetwork`. A curator cannot tell whether a scope is near its cap, so the feature needs a new read/maintenance surface (§4). 3. **No scope enumeration.** `e.stores` is keyed by `scope.Key()` and nothing lists scopes. `sessioncurator.Pass` derives scopes only from `Conversations.RecentSessions`, so the curator can only ever touch agent-user scopes of recently active sessions. Cold scopes, agent scopes and global scope are out of reach (non-goal §7). 4. **`retireOldest` marks rotation as a deletion today.** It calls `appendHistory(..., true)`, setting `markerData.Deleted` — the same flag `Forget` sets, with no way to tell them apart. The issue's request (a `rotated` flag) is real and also **fixes a mislabel**: a rotated record was not deleted by anyone. 5. **AC3 ("a pass with no model available still succeeds via rotation") cannot happen as wired.** `archiegateway.setupCurators` returns `false` and registers *no* curators when `chatModels.ModelFor` fails ("curators disabled: chat model unavailable; configure a chat model and restart the gateway"), and a provider failure at pass time propagates out of `reviewOne` and fails the whole pass. Either AC3 means "a pass where the *compaction* model call fails", or curator startup has to change for the whole family (skill curator and definition engine included). §6 adopts the first reading and §9 flags it for confirmation. 6. **The "cheaper, model-free coalescing" alternative does not work.** Merging N note blocks into one "earlier observations" record changes record *count*, not *bytes* — the document is exactly as full afterwards, so it never makes room and rotation still fires at the cap. Only lossy compression reduces occupancy. The model call is **load-bearing**, not an optional upgrade. (The one thing deterministic merging would fix is the 20-record window; a summary fixes both.) ## 3. Goals - A scope above a size threshold gets its oldest records replaced by one smaller summary record during a session-memory curator pass. - The replaced originals are retained in `HISTORY.md` and are distinguishable from `Forget` deletions there. - A failed or unavailable summary degrades to today's behaviour: no write, no failed pass, rotation still bounds the document. - No new configuration surface, no new service, no new process. ## 4. Proposed solution ### 4.1 Domain contract (`internal/domain/memory/contract.go`) — one new operation ```go // Summarizer compresses a scope's oldest records into one replacement // record's content. The curator supplies the model call; the engine owns // where and when it is applied. type Summarizer func(ctx context.Context, contents []string) (string, error) // Compact replaces q's oldest records with one smaller record when the // scope's live document is near its bound. Below the threshold it is a // no-op with a zero result and no call to summarise. Compact(ctx context.Context, scope Scope, summarise Summarizer) (CompactionResult, error) type CompactionResult struct { OccupancyBefore, OccupancyAfter int // bytes of the live document SummaryID RecordID Retired []RecordID } ``` Added to `Store` (not an optional interface): `BuiltinEngine` is the only implementer — verified, no other type implements `Manifest() domainmemory.Manifest` and no fake implements the full `domainmemory.Store` (the gateway's narrowed interfaces are unaffected). The engine keeps the format knowledge, the atomic ordering and the threshold; the curator contributes only the model call. It also means the curator needs no occupancy query: below the threshold `Compact` returns immediately without calling `summarise`. Also on the contract: `Revision` gains `Rotated bool` beside `Deleted bool`. ### 4.2 Builtin engine (`internal/infrastructure/memory/builtin_engine.go`) Algorithm for `Compact`: 1. Resolve the scope's stores read-only; no directory → zero result. 2. `stores.live.Reload()` first, so a write made by another process holding the file is visible. 3. Occupancy = `len(stores.live.Render())`; if `occupancy < compactThresholdRatio (0.8) * maxBytes` → return zero result, nil. 4. Candidates = live blocks sorted by `UpdatedAt` **ascending**, taken inline-document order for ties, up to `compactMaxCandidates` (32) or `compactBudgetBytes` (8 KB of content), whichever comes first. Every kind is eligible, including a previous summary (gradual forgetting is intended; the ≤50%-smaller guard bounds the damage). Newest records are never candidates. 5. Call `summarise(ctx, contents)`. Error → return it **without writing anything**. 6. Size guard: if `len(summary) >= 0.6 * (bytes of candidates replaced)`, write nothing and return an error ("summary did not make room"). This is what stops a "summary" that is not a win from retiring records for nothing. 7. Write order, chosen so a crash never leaves a fact counted twice: a. `appendHistory(..., rotated=true)` for each candidate; b. `stores.live.Remove(...)` each candidate — a missing block is success, not an error (another process may have rotated it); c. `stores.live.Add("summary", renderBlock(...))` the summary **last**, so the file has room by construction and `Add` cannot re-enter rotation. A crash between (b) and (c) leaves the originals retained in history with no summary — exactly today's guarantee, not a new loss. 8. Summary record provenance: `Kind: "summary"`, `Author: "session-memory curator"`, `Source: "compaction of N records"`, `Revision: 1`. It lands in a `## summary` section (valid under `builtin.ValidateSectionName`); it is the newest record, so it sits at the top of the 20-record read window. Concurrency: `BuiltinEngine.mu` guards only the store map and `builtin.Store` locks per operation, so a compaction is not atomic against a concurrent chat-turn `Create`. The `Reload` + missing-block tolerance above is the mitigation; a concurrent write in the middle can cost one duplicate summary line, never a lost record. ### 4.3 History marking (`history.go`, `marker.go`) - `markerData` gains `Rotated bool \`json:"rotated,omitempty"\``; `revisionsOf` maps it into `Revision.Rotated`. - `appendHistory` takes an explicit reason instead of the lone `deleted bool` — e.g. `(deleted, rotated bool)`: - `Update` → `(false, false)` - `Forget` → `(true, false)` - `retireOldest` → `(false, true)` ← **behaviour change**: rotation is no longer labelled a deletion - `Compact` → `(false, true)` - No backfill: pre-existing rotated entries keep `deleted: true` because nothing recorded which reason applied. Harmless — no reader today — and stated so the reviewer is not surprised by mixed markers. ### 4.4 Curator (`internal/infrastructure/sessioncurator/sessioncurator.go`) - `Pass` collects `map[domainmemory.Scope]struct{}` of every scope it considered while iterating sessions (built before the model call in `reviewOne`, so a skipped or empty session still contributes its scope). - After the extraction loop, for each scope (bounded to `maxCompactionsPerPass` = 3, deterministic order by scope key): `engine.Compact(ctx, scope, c.summarise)`. - `c.summarise` is one model call with a new `compactPrompt` ("merge these observations into one record of at most X words, dropping anything superseded, dated or one-off; keep names and preferences verbatim"), reusing `c.host.LLM` / `c.host.Model` exactly as `askModel` does. - Error isolation: **any** error from `Compact` (model failure, no room made, engine error) becomes an action, not a `Pass` error: - `ActionCompacted = "memory.compacted"`, detail `"scope <kind>: <n> record(s) → 1 summary, <before> → <after> bytes"`; - `ActionCompactionSkipped = "memory.compaction_skipped"`, detail = the error, reason = why. - Extraction failures keep their current semantics (a real provider failure still fails the pass); the change is scoped to compaction. This keeps a compaction problem from ever costing a pass's extracted observations. - Budget: a pass makes at most 3 compaction attempts, and zero model calls when no scope is above the threshold, because the engine short-circuits. That matters because passes share a single 10-minute slot process-wide (`defaultPassTimeout`, `MaxConcurrentPasses` default 1). ### 4.5 The reader for the rotated flag Storing the flag alone would be "a field with no consumer" — AGENTS.md's complaint. Two readers, no new UI: 1. `Revision.Rotated` through the existing `Revisions` contract method (engine-level verification, and the seam any future history viewer uses). 2. **User-visible**: the `memory.compacted` / `memory.compaction_skipped` curator actions appear in `CuratorHealth.recent_actions` (`proto/gateway/v1/chat.proto`) and render on the existing dashboard page `/settings/curators` (`ui/src/curators/CuratorCard.vue`, ring buffer of 20 in `domain/curator/activity.go`). That is the human-facing "why did these records leave". A `HISTORY.md` / memory browser is a non-goal (§7) and a separate issue; there is no memory page in the dashboard at all today. ### 4.6 No new configuration Deliberately no `max_file_bytes` or `compact_threshold` field. `MemoryConfig` carries only `Engine`, `config.example.toml` has no `[memory]` block, and `controlplanestore` has **no memory settings kind** — a config field would need a new control-plane kind and dashboard form to satisfy "a setting must work end to end". Instead the engine reads its own cap (`Store.MaxBytes()`) and applies named constants (`compactThresholdRatio = 0.8`, `compactMaxCandidates = 32`, `compactBudgetBytes = 8 KB`, `maxCompactionsPerPass = 3`). If tuning becomes necessary, that is a separate `memory-settings` control-plane issue. ## 5. Files and components affected | File | Change | Est. lines | |---|---|---| | `internal/domain/memory/contract.go` | `Summarizer`, `CompactionResult`, `Store.Compact` doc+signature, `Revision.Rotated` | ~45 | | `internal/infrastructure/memory/builtin_engine.go` | `Compact` + candidate selection + constants | ~110 | | `internal/infrastructure/memory/history.go` | `appendHistory(deleted, rotated)`, `revisionsOf` maps `Rotated` | ~12 | | `internal/infrastructure/memory/marker.go` | `Rotated` field | ~4 | | `internal/infrastructure/memory/builtin_engine.go` (`retireOldest`) | pass `rotated` instead of `deleted` | ~2 | | `internal/infrastructure/sessioncurator/sessioncurator.go` | scope collection, `summarise`, `compactPrompt`, 2 action types, error isolation | ~90 | | `internal/infrastructure/memory/builtin_engine_test.go` + a sessioncurator test | new cases (table-driven) | ~120 | | Dashboard / proto / config / gateway wiring | **none** | 0 | ## 6. Acceptance criteria Test seam changes here (the memory engine contract and provenance semantics), so tests are warranted per AGENTS.md. 1. **Compaction happens.** Given a scope whose `OBSERVATIONS.md` is at ≥80% of its cap, one session-memory curator pass over that scope replaces its oldest records with exactly one live `summary` record, and the scope's live byte count falls. Verified by a table-driven engine test with a stub summariser and a fixed `maxBytes` (following `TestBuiltinEngineCreateRotatesFullSection`'s `NewBuiltinEngine(t.TempDir(), 4096)`). 2. **Originals are retained and marked.** Each replaced record returns `Revisions` entries with `Rotated == true` and `Deleted == false`, content unchanged; a record removed via `Forget` still returns `Deleted == true, Rotated == false`; an `Update` produces neither flag. 3. **Rotation is untouched as the fallback.** `TestBuiltinEngineCreateRotatesFullSection` still passes unfiltered, and now also asserts `Rotated == true` on the retained record. 4. **Degradation is fail-soft.** With a summariser that returns an error, and separately one that returns a summary larger than 60% of its input: `Compact` returns that error, `OBSERVATIONS.md` and `HISTORY.md` are byte-identical to before the call, the curator pass returns a `memory.compaction_skipped` action and **no error**, and further `Create` calls still succeed (rotation still fires at the cap). 5. **Below the threshold is free.** `Compact` on a scope under 80% returns a zero result, writes nothing, does not call the summariser, and the pass makes no extra model call — assert the stub summariser's call count is 0. 6. **No new configuration exists.** `config.example.toml` and `MemoryConfig` are unchanged; a reviewer can confirm `git diff --stat` touches no file under `internal/config`, `proto/` or `ui/`. ## 7. Explicit non-goals - No history/memory viewer UI; no `HISTORY.md` reader beyond `Revisions`. Follow-up issue. - No scope enumeration (`Scopes()` on the engine). Scopes with no recently active session are not compacted; rotation remains their bound. Follow-up if cold scopes matter. - No change to curator startup without a chat model (§9, Q1). The skill curator and definition engine would be affected, so it is a separate issue. - No change to the 100 KB live cap, the 4 MB history cap, `memoryRecordsPerScope` (20) or the 8192-byte prompt block cap; no ranked retrieval. - No id linkage from a history entry back to the summary that replaced it (`replaced_by` in the marker). ~3 extra lines if the reviewer wants it; recommend deferring, since nothing reads it. - No record pinning / "never summarise this"; no per-scope policy. - No migration or rewrite of existing `OBSERVATIONS.md`/`HISTORY.md` files. - Not a fix for a lost-work bug: rotation already bounds loss, hence P3 and no place in the 1 December push. ## 8. Risks - **Summary quality is a prompt problem with a size guard, not a correctness guarantee.** A bad summary can drop a fact that a turn would otherwise have read. Bounded by the 60%-smaller guard and by history retention; accepted for a solo-instance product. The alternative (rotation) is strictly worse for the model's view. - **Summary-of-summary degradation** across cycles. Accepted; each cycle must halve to be accepted, so it is slow. - **Non-atomic against concurrent writers** (§4.2). Mitigated by `Reload`, missing-block tolerance and summary-last ordering; worst case is a duplicated line. - **Extra model spend**: at most 3 calls per hourly pass, zero when no scope is near its cap. Notes accumulate ~5 facts × ~250 bytes per session, so one 32-candidate compaction (~8 KB → ~3 KB) outweighs a typical hour of growth per scope. ## 9. Two decisions I need confirmed (recommendations given) 1. **AC3 as written is unachievable.** "A pass with no model available still succeeds via rotation" — there is no pass without a model: `setupCurators` registers no curators at all when `chatModels.ModelFor` fails. I recommend restating it as AC4 above ("a pass whose compaction model call fails still succeeds, and rotation still bounds the document"). If you want the literal reading, the change is in `archiegateway.setupCurators` plus a nil-LLM guard in the session-memory curator, and it also changes skill-curator and definition-engine startup — a separate issue, and I would not bundle it here. 2. **Is the model call required, or is any size-reducing replacement acceptable?** I recommend required (it is the only thing that reduces bytes, §2.6), with the guard in §4.2/6 that a non-shrinking result is rejected rather than applied. ## Estimated diff size ~265 lines of production code + ~120 lines of tests ≈ **385 lines** across 6 files in 3 packages (`domain/memory`, `infrastructure/memory`, `infrastructure/sessioncurator`), zero changes to proto, config, dashboard or the composition root. Well past AGENTS.md's ~100-line "stop and assess" mark, which is why this is a PRD and not a patch. ## Verified vs not verified Verified by reading the tree: every code reference, constant and call site above; the AC3 contradiction; the absence of a control-plane memory settings kind; `Revisions` having only one (test) caller; `BuiltinEngine` being the sole engine implementation; no fake implementing the full `domain/memory.Store`. Not verified: no code was run and no test executed (`task check` not run); the ~260-byte note-block estimate and the 0.8/32/8 KB tuning values are arithmetic from the marker format, not measurements; whether the operator wants `Rotated` surfaced anywhere beyond the curator actions is their call (§9 Q1 is the only blocking one). _Decide in chat or on the dashboard._
sam added the deferred label 2026-11-15 17:26:14 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1669