Files
archie/docs/architecture/migration-decisions.md
T
sam a5d3472b4b feat(memory): builtin engine implementing the domain/memory contract
Part of archie-core-1786637499480 (ship the file-writing store as the
default memory engine).

Records the missing design decision first (migration-decisions.md#5,
"authoritative memory record and revision model" was explicitly open):
internal/memory/builtin.Store addresses blocks by content substring with
no stable per-block ID, which the new Record{ID}/Forget(id) contract
needs. Decided and implemented: an ID assigned once at Write via a
leading `<!--mem:<uuid>-->` marker line, stripped back out of
Record.Content on read, located by that marker (not content) for
Forget -- so an unrelated edit can never orphan a Forget. Identity maps
to a directory named hex(sha256(identity)), not the identity string, so
an arbitrary identity can never traverse outside the configured root.
Kind maps to a builtin section name via the store's own
ValidateSectionName, so an invalid Kind fails the same way an invalid
section already does.

internal/infrastructure/memory.BuiltinEngine wraps one builtin.Store per
identity and implements domain/memory.MemoryEngine end to end
(Write/Query/List/Forget, Lifecycle, Manifest). Lives under
internal/infrastructure per organisation.md ("infrastructure implements
domain contracts"), not internal/domain/memory, which stays
contract-only.

internal/memory/builtin is untouched -- confirmed via git status and its
own test suite passing unmodified, per this issue's acceptance
criterion. This engine is not yet wired into daemon bootstrap or the
config surface (archie-core-1786637490043); it proves the contract
holds for the builtin store the same way curator's reference
implementations prove the curator contract, and registers/runs cleanly
through domain/memory.Registry (TestBuiltinEngineRegistersAndRuns
ThroughTheRegistry).
2026-09-02 02:36:10 +10:00

17 KiB

Architecture Migration Decisions

Status: Active migration inventory
Date: 2026-07-28
Tracking issue: #73

Purpose

This document records what remains to be decided before Archie's current implementation can be migrated into the approved domain-oriented structure.

The product and core domain model are not being redesigned. Existing Archie behaviour and the existing session implementation are the migration baseline. Remaining decisions MUST be grounded in current packages, consumers, tests, persistence, and runtime wiring.

Confirmed domain packages

internal/domain/
  identity/    (approved, not yet materialised as a package)
  agent/       (approved, not yet materialised as a package)
  workflow/
  messaging/   (canonical Message/Conversation types only; gateway migration pending, see section 2)
  workintake/
  plugin/      (approved, not yet materialised as a package)
  curator/
  mapping/
  scheduling/
  • identity owns persistent actors, lifecycle, ownership, attribution, and audit identity.
  • agent owns persistent assistants, specialisation, instructions, model-independent continuity, capability context, and associations with identity and memory.
  • workflow is a separate domain. It owns reusable, versioned Workflow definitions, WorkflowSteps, WorkflowExecutions, StepExecutions, Attempts, outcomes, and typed Workflow plugin contracts.
  • messaging owns canonical Messages, Conversations, branches, channel and source correlation, replies, acknowledgements, delivery attempts, events, and projections.
  • workintake owns admission, validation, deterministic routing, and accepted work requests for the optional transition into durable workflow-backed execution.
  • plugin owns generic plugin identity, discovery, compatibility, and lifecycle mechanics. It remains metadata-only; capability semantics and typed registration belong to the owning domain.
  • curator owns the curator engine family: long-running agent loops that reason over memory and perform maintenance tasks on a declared shape (check-in interval, tool set, attached memory engine). See epic archie-core-yp9.
  • mapping owns the payload field-mapping vocabulary: binding named fields to JSON paths inside a captured webhook payload, and detecting drift. See docs/prds/payload-field-mapping.md.
  • scheduling owns the ticker engine: deciding when a scheduled job runs and with what concurrency. It declares JobSource/Runner contracts and owns no storage or delivery. See epic archie-core-1786637496320-249-be6d9a20.

Reconciliation (2026-08-31): curator, mapping, and scheduling existed in the tree without being added to this list when their epics landed. They are confirmed as legitimate, narrowly-scoped domains under the same rules as the original six — each owns one capability family, declares typed contracts, and has no cross-domain reach. This list was out of date, not the tree.

Conversely, identity, agent, messaging, and plugin are approved target domains that do not yet exist as packages. Their behaviour currently lives in pre-migration packages (see "Complete package destination map" below for the full inventory). Do not create empty stub packages for them ahead of an actual migration PR — an unused stub is dead code and gives no test coverage. Each materialises when its corresponding migration decision below is resolved and implemented in the same change.

Memory's final package placement remains undecided pending focused review of the existing implementation.

The following are approved shared or composition packages, not domains:

internal/eventbus/
internal/policy/
internal/infrastructure/
internal/app/archied/
internal/app/agentworker/

Remaining migration decisions

1. Complete package destination map

Every current package MUST receive one explicit disposition:

  • move into a confirmed domain;
  • move under infrastructure as an implementation of a domain-owned contract;
  • move into application composition;
  • merge with an identified duplicate;
  • remain as an approved shared application contract;
  • move to a plugin, extra, deployment, example, or generated artifact;
  • be deleted after its replacement is proven.

The largest unreviewed areas include memory, tools and capabilities, workspace and worktree behaviour, gates, storage, events, web UI, containers, RPC packages, release management, and scheduling.

The destination matrix MUST name the behaviour owner, target path, dependencies to remove, state and contracts to preserve, migration prerequisites, and deletion criteria.

2. Session and Messaging migration

The existing session implementation is the migration baseline. The migration must decide:

  • which session and message components are retained or adapted;
  • how Agent, user, channel, Conversation, and branch ownership are added;
  • how Archie's parallel gateway session implementation is replaced;
  • how canonical Messages become immutable persisted records;
  • how existing sessions and messages are migrated;
  • how outbound delivery, retry, acknowledgement, deduplication, and failure semantics are represented;
  • how current Telegram, email, webhook, forge, and Jira integrations migrate to Messaging infrastructure adapters.

The following are already decided and MUST NOT be reopened without contradictory implementation evidence:

  • the composite Conversation identity;
  • immutable MessageIDs;
  • typed tool-call and tool-result messages;
  • exact branch lineage through ParentConversationID and ForkMessageID;
  • parent and child transcript isolation;
  • normal scoped-memory behaviour regardless of the originating Conversation;
  • Messaging and Work Intake ownership.

Decision (2026-08-31), resolving the open questions above:

internal/gateway/ is the "parallel gateway session implementation" this section names. It is retained as-is for now — a big-bang rewrite of live session/compression/approval/branch logic across Telegram, web UI, email, and webhook channels in one change is exactly the "convoluted to satisfy edge cases" failure mode CLAUDE.md's Revert-on-Flail rule warns about, and would risk silent behavioural regressions in the one path every user-facing message travels through.

Instead, internal/domain/messaging/ is materialised now with the pieces that are already decided and don't require touching gateway: the canonical Message and Conversation types (composite Conversation identity, immutable MessageID, typed tool-call/tool-result message variants, branch lineage via ParentConversationID/ForkMessageID). These are pure domain types with no gateway dependency — a new package, not a rewrite of an existing one.

Migrating internal/gateway/'s session, compression, approval, and branch logic onto these types is real, cross-channel-risk work and is not done in this pass. It is tracked as separate, appropriately-scoped bd issues (see bd list --label messaging-migration), sequenced channel-by-channel so each lands with its own behavioural verification instead of one large diff no one can safely review.

3. Identity data migration

The migration must define:

  • the durable identity schema and repository implementation;
  • bootstrap of the permanent system identity;
  • stable IdentityIDs for existing configured Agents;
  • migration from configured identity names, empty-string ownership, and bot usernames;
  • preservation of historical attribution;
  • identity-aware uniqueness and transport deduplication;
  • correction of container and RPC capability resolution so execution uses the owning identity's bindings;
  • removal of identity-string equality from access-control behaviour.

Exact schema and method signatures are implementation decisions. The approved Identity lifecycle and attribution semantics are fixed migration constraints.

4. Workflow migration

The current implementation must be transformed deliberately:

store.Task            -> WorkflowExecution
mutable Stage         -> StepExecution history
RetryCount / Attempt  -> Attempt records
workflow.Registry     -> versioned Workflow definitions
current stages        -> WorkflowStep definitions or Workflow plugins

The migration must decide:

  • Workflow definition and version persistence;
  • the smallest accepted-work-request contract consumed from Work Intake;
  • the enforced WorkflowExecution state machine;
  • atomic state and domain-event persistence;
  • migration of existing task rows and transition history;
  • disposition of current skill, Yaegi, and stage-plugin implementations;
  • the versioned worker-handoff contract;
  • scheduler, claim, retry, crash-recovery, and waiting-human cutover.

Workflow remains a separate domain from Agent. A user or Agent may define or invoke a Workflow, but Workflow semantics and plugins remain Workflow-owned.

5. Memory placement and storage

The four required scopes are fixed:

  • global shared;
  • Agent-wide;
  • user-wide;
  • Agent-user relationship.

A focused review of internal/memory, its providers, consumers, persistence, and tests must decide:

  • internal/domain/memory versus ownership within the Agent domain;
  • the authoritative memory record and revision model;
  • provider and infrastructure boundaries;
  • retrieval and access enforcement;
  • migration of existing memory data;
  • provenance representation.

Conversation branches do not create another memory scope. A memory action has the same scoped effect regardless of which Conversation originated it.

internal/domain/memory ownership — DECIDED (2026-09-02). The typed engine family lives at internal/domain/memory (archie-core-1786637490069-11-e245c170), mirroring internal/domain/curator's contract/registrar/registry split. Not owned within the Agent domain: memory is consumed by more than agent turns (curators, per archie-core-1786637499636) and the plugin engine rule requires its own family, not a capability bolted onto another domain's contract.

Authoritative record and revision model for the builtin engine — DECIDED (2026-09-02), scoped to unblock archie-core-1786637499480 (ship the file-writing store as the default engine). This decides only the builtin engine's mapping onto the new Record{ID}/Forget(id) contract, not the memory model in general or a future non-file engine's model:

  • internal/memory/builtin.Store is unchanged. It has no concept of a stable per-block ID — blocks are addressed by content substring — and nothing in this decision requires changing that; existing internal/memory tests keep passing unmodified, per that issue's acceptance criteria.
  • A new infrastructure adapter (internal/infrastructure/memory, per organisation.md's "infrastructure implements domain contracts" rule — not under internal/domain/memory, which stays contract-only) wraps one builtin.Store per identity.
  • Identity → directory: hex(sha256(identity)), not the identity string itself. Closes the path-traversal question without an allowlist regex, and sidesteps collisions from identities that differ only in characters a filesystem treats as equivalent.
  • Record ID: assigned once at Write, independent of content, so a later edit elsewhere in the file can never orphan a Forget. Realized as a leading marker line (<!--mem:<ulid>-->) on the block builtin.Store.Add writes; Query/List strip it back out of Record.Content before returning, and Forget(id) locates the block by that unique marker via the store's existing substring-based Remove. The marker is a pragmatic, reversible choice — plain-text records stay human-readable and human-editable, at the cost of one marker line an operator editing the file by hand should leave alone. Revisit if/when provenance representation (still open, below) settles on something richer.
  • Kind → section: Observation.Kind (default "general" when empty) is the section name, reusing builtin.ValidateSectionName — so an invalid Kind fails Write with the same validation error memory_edit already returns for an invalid section today, not a new error shape.

6. Runtime and process boundaries — DECIDED

archied is the native resident orchestration and interactive-chat process. Every autonomous repository workflow crosses one full-task core-NATS handoff into a task-scoped archie-agent container. Embedded versus external NATS changes broker deployment only; it never selects an executor. The former host in-process runner, per-stage NATS protocol, and subprocess runner are removed.

Forge, store, and worktree authority remains in archied and is exposed to the task container through scoped RPC services. The worker runs the workflow and its worker-local ai-sdk loop; it does not receive forge credentials or direct store access. See docs/prds/embedded-nats.md.

7. Shared mechanics

The following cross-domain mechanics require exact contracts before dependent packages migrate:

  • event-bus delivery and failure semantics;
  • policy evaluation API, composition, and precedence;
  • configuration candidate preparation, health validation, atomic promotion, observation, and rollback;
  • service-specific health, bounded remediation, isolation, and escalation;
  • changelog association and grouping.

These shared mechanisms do not acquire ownership of domain meaning. Domains continue to own their commands, events, policies, settings, and consequences.

The copied Go specgen and VitePress documentation pipeline, output placement, developer commands, drift checking, hook sequence, and CI sequence are defined in generated-documentation.md. Adapting the copied files, implementing Archie's normalized documentation model and page renderers, and cutting docs over to its final location remain migration work rather than open product architecture.

8. Cutover sequence

The final migration plan must order:

  • compatibility adapters;
  • schema migrations and data backfills;
  • any required dual-read or transitional paths;
  • domain-by-domain package replacement;
  • production wiring changes;
  • legacy package removal;
  • architecture tests preventing dependency regression;
  • focused, integration, race, and full quality gates.

Each legacy package requires objective removal criteria. No compatibility path may become an indefinite second implementation.

Completion criteria

Migration design is complete when:

  • every current package has one approved owner and destination;
  • every mutable record has one authoritative owner and migration path;
  • all external and cross-process contracts have versioned owners;
  • existing behaviour and data have explicit parity requirements;
  • process boundaries have evidence-backed justification;
  • the cutover order preserves a working application;
  • legacy deletion criteria and deterministic architecture tests are defined;
  • no unresolved decision requires an implementer to invent domain semantics.

Exact Go APIs, SQL statements, and mechanical refactoring steps MAY be finalized during implementation when they do not change these semantics.

Recorded migration position (2026-07-29)

Recovered 2026-08-09 from the pre-migration issue tracker. This section records decisions the maintainer approved that the repository did not otherwise capture.

Completed: internal/infrastructure/eventbus/nats (rebuilt, not relocated); internal/infrastructure/configuration (loading moved out of internal/config, which is now types-only and does no I/O); internal/eventbus (broker-neutral contract); internal/domain/workintake (the first internal/domain/ package, owning TaskEnvelope, the routing Kind, the label vocabulary, and task subjects).

DECISION — private input DTO deferred (approved). configuration.Document.Config is still internal/config.Config rather than the private per-domain input DTO this document mandates. Building an ~800-line parallel type tree while every consumer still reads config.Config would mean two shapes kept in sync by hand, with no consumer for the new one. It is named as a field so the compromise is visible at each use site, with the deletion condition recorded in that package's doc.go. Do this with the internal/config dissolution, not ahead of it.

Agreed next: internal/app/{archied,agentworker}. run() in cmd/archied/main.go is ~460 lines, against organisation.md's rule that cmd/* must not contain substantive wiring or act as a service locator. It is also the last non-infrastructure holder of the NATS SDK, and it is where the config DTO translation will land, so it unblocks the dissolution above.

Method constraint. Do not write a separate recorded package-review document per area when the decisions are already dictated by dependencies-and-contracts.md. That is the ceremony pattern this project deliberately avoids.