Commit Graph
21 Commits
Author SHA1 Message Date
sam da28ddb5e6 refactor(ui): sever the archie-ui binary from every daemon runtime package
Completes the objective deletion gate for the UI Service
(archie-core-8cda.5.6): go list -deps ./cmd/archie-ui now links zero
banned packages -- no modernc.org/sqlite, no internal/store, gateway,
channels, memory, secret, agentexec, tools, skill, yaegiutil, curator,
or the workflow engine. The closure shrinks from 68 to 26 archie-core
packages. Each cut is a producer-side move; no consumer-side defensive
shims.

- Store contracts (TaskStore family, value types, sentinels) moved from
  internal/store to internal/domain/storecontract -- the approved contract
  location per state-store-contract rev. 2d, and NOT internal/contracts:
  that tree is buf generate output and tools/proto.sh check deletes
  hand-written files there. internal/store keeps aliases; sentinel
  message strings are byte-identical and remain the wire contract.
- Chat contracts (ChatContract, sessions, turns, streaming, commands)
  moved from internal/gateway to internal/domain/messaging; gateway keeps
  aliases. ToolCallEvent rendering helpers become gateway functions
  (methods cannot live on alias types).
- Task vocabulary (Task, statuses, mid-run Store, Definition) split into
  internal/domain/workflow/task; the pipeline engine stays in package
  workflow with aliases, so daemon-side callers are unchanged.
- overlay.DeniedKeys moved to internal/infrastructure/configuration:
  webui referenced the constant and that dragged the overlay SQLite
  store in.
- webui.Server's concrete runtime handles become webui-owned view
  interfaces with daemon-side bootstrap adapters: channel lifecycle
  (internal/channels/status), MemoryStatus, CuratorStatus, and a
  SkillCatalog seam replacing the direct skill.DefaultRoots/CatalogRoots
  calls. Each degrades to an empty page when unwired.
- config.SecretRef moved into internal/config (config vocabulary);
  internal/secret keeps an alias and Registry.Resolve absorbs the
  zero-value short-circuit.

Docs: state-store-contract rev. 2d relocation record; ui-service-boundary
deletion gate marked satisfied; ARCHITECTURE.md package map rows.
2026-09-10 03:03:33 +10:00
sam 7c8f625f0e feat(gateway): flip channel boundary onto messaging.Message
Replaced gateway.Message as the currency at the channel-facing boundary
with the canonical internal/domain/messaging.Message. Channel adapters
now build the record directly (RoleUser, ConversationID, Sender) and the
Router hands it back unchanged, so nothing converts between two shapes in
process. gateway.Message is deleted; the dashboard route that had no home
in the record lives in a new Inbound envelope (Message + transport-only
Page), which reaches the system prompt and is never persisted.

Role is now set at construction instead of derived at the store boundary,
and the assistant reply is a direct messaging.Message (RoleAssistant,
Sender=botUser). The gRPC proto is unchanged: pb.Message still carries no
role, and both sides keep deriving it from the owning session's bot
identity via gateway.RoleForSender (ToStoredMessage/FromStoredMessage are
deleted).

Adds tests that the old shape could not express: Page reaches the system
prompt but not the persisted record, and a channel-built Inbound round-trips
with the Role the channel set.

Refs: archie-core-9g5e
2026-09-09 01:05:36 +10:00
sam 571323138d feat(messaging): migrate gateway session persistence onto domain types
SessionStore message history now stores/loads domain/messaging.Message:
sender attribution, derived role, and conversation address are canonical
record fields. SQLite gains a role column with open-time backfill for
pre-migration databases. gateway.Message remains the channel-facing
transport; conversion lives in one boundary module. ChatContract,
gatewayrpc proto, and dashboard JSON are unchanged; the RPC client
splits into ChatContract (Client) and SessionStore (StoreClient) views.

Sessions stay SessionContext: UUID keys and platform/bot/title/branch
metadata have no lossless home in Conversation; keyed migration tracked
separately once Agent/user/binding ownership settles.

Refs: archie-core-d9dv
2026-09-08 21:28:19 +10:00
sam 1f71489f0a fix(gateway): harden session recovery paths
Repair future-dated conversation history on SQLite open and cap newly supplied timestamps so one bad platform clock cannot pin a session indefinitely.

Make message and turn identifiers injective while preserving legacy redelivery and crash-recovery compatibility, retire idle turn lanes safely, and pin the existing compression and partial-undo behavior with focused regressions.
2026-08-09 23:18:55 +10:00
sam 61b4b7b0cf refactor(store): replace legacy persistence with SQLite
Task and event persistence now uses a dedicated SQLite sidecar derived from the configured state prefix, matching the independently persisted conversation store. The configured path now means a state-path prefix rather than a database file so fresh SQLite state never attempts to interpret the previous storage format.

Remove the retired implementation and dependency completely, strengthen SQLite migrations and concurrency behavior, restore backend-neutral coverage, and align configuration, operational guidance, UI labels, and deployment documentation with the new storage and host-daemon topology.
2026-08-09 22:27:25 +10:00
sam b876fc80cc feat(gateway): make chat turns durable and idempotent 2026-08-09 01:15:41 +10:00
sam fa6f4aef5f fix(gateway,daemon): repair regressions found by the adversarial review
Six defects, four of them introduced by the fixes in dbacb96, 44d43ef,
d5e016e, de1a899 and 4d44ef7. This is the third round in this codebase
where a round of fixes broke something the next round caught.

SANDBOX SILENTLY DROPPED. startContainers degrades a failed container
pool to nil, and the daemon's only test was `if ContainerPool != nil`.
So `containers.enabled = true` plus a missing agent image ran every
task's agent loop in-process, with the daemon's own authority and the
host filesystem -- the one thing [containers] exists to prevent -- while
logging only that containers were disabled. Nil now means two things and
they are distinguished: SandboxRequired parks the task instead. The
daemon still stays up; chat and the dashboard are unaffected.

SUMMARY AT THE WRONG END OF HISTORY. compressedHistory returns [head,
summary, tail], but ordering is by timestamp, not slice position. The
retained messages keep their original timestamps -- that is what
preserves their identity -- so the summary, being new, was clamped to
now and sorted after the tail it was meant to precede. The model read
the recent conversation and was only then told earlier context had been
summarised. Confirmed on both backends: index 23 of 24.

The monotonic clamp belongs to the append path, where an inbound message
must not sort ahead of the one that provoked it. ReplaceMessages is a
rewrite primitive and now honours explicit placement, and the summary
takes a timestamp inside the span it replaces.

A COMPRESSION SUMMARY ANSWERED A QUESTION. Following from the above,
PriorReply saw a bot-attributed record immediately after an unanswered
message and replayed it as the answer: a redelivered question was
"answered" with the compression banner and the model never ran. The
user's question was silently swallowed.

RECENCY STILL DIVERGED. sessionRecency preferred LastActiveAt whenever
set; SQLite used MAX(last_active_at, created_at). Those agree only when
LastActiveAt is the later of the two, so a clock step or an older record
made the caller and the store order the same sessions differently.

LASTACTIVEAT WAS NEVER UPDATED. Touch had no caller, so "resolve the
most recently active session" meant the most recently created one -- an
operator who moved to an older session with /topic and worked there was
dropped back into the newer, abandoned one on restart, which is the bug
d5e016e claimed to fix, with the direction reversed.

/branch WITH NO ACTIVE SESSION PANICKED, slicing an empty parent ID for
its default title. Pre-existing, but reachable: email and webhook call
Route directly without resolving first, and the email path runs in a
goroutine with no recover, so one message took the daemon down.

The sandbox guard has no test. The honest reason is that exercising it
needs a daemon with a forge and worktree the current harness does not
build, and a test that re-states the condition proves nothing -- which
is the same false confidence that let these regressions through.
2026-08-05 04:24:16 +10:00
sam a1a111ff35 fix(gateway): give branch-inherited messages their own identity
/branch copies the parent's history into the child. Because the store
honours an incoming MessageID -- which is what lets a read-modify-write
preserve identities -- the copies kept the parent's canonical IDs, so
two sessions claimed one identity and branch-point correlation was
ambiguous. Two branches from one parent made it three. It also broke
dedup in the child: an inherited message carried an ID derived from the
parent's session, so redelivering it appended (confirmed on both
backends: 3 messages became 4).

newMessageID's doc already claimed the opposite behaviour, and the
Message.MessageID doc said it was "ignored on write" -- which is exactly
the misreading that produced this. Both now describe what the code does,
including the obligation on a caller copying messages between sessions.

Also removes a doc comment describing msgID and its disambiguating
counter, deleted in 0814f7b, which had come to sit above
messageIDNamespace describing the opposite of the current design; and
corrects the Telegram comment still claiming a redelivered update
"updates the stored turn", the behaviour the immutability fix removed.

Refs archie-core-la1
2026-08-04 22:25:20 +10:00
sam f64353f0b5 fix(gateway): do not answer a redelivered message twice
Making the store idempotent stopped redelivery duplicating the inbound
record, but nothing consulted that result: respond() saved the message,
ignored the fact that nothing had changed, and called the model anyway.
The cost was a second billed turn, any tool side effects run a second
time, and a second assistant reply appended -- replies carry no upstream
reference, so they always get a fresh identity and always append. The
stated problem, "a re-delivered update appends a duplicate turn", was
only half fixed.

PriorReply answers "have I already replied to this?" from the history
the caller has already loaded, and the chat path replays that answer
instead of regenerating it. A message with nothing after it was never
answered -- the turn may have died partway -- so that still runs, since
the user is owed a reply.

CanonicalMessageID exposes the store's own derivation rather than
letting the caller reimplement it, since a lookup that silently never
matches is indistinguishable from a message never seen.

Refs archie-core-i6q
2026-08-04 22:23:17 +10:00
sam 0fd3398cf8 fix(gateway): stop /compress destroying a message that arrives midway
applyCompress reads the history, computes a summary, then replaces it.
ReplaceMessages removed everything outside the replacement set, so a
chat turn saved in that window was deleted without ever having been
read, let alone summarised. Confirmed on both backends. It is
unrecoverable loss, and it sits outside the write-then-delete guarantee
because the window is in the caller, not the store.

ReplaceMessages now takes the messages the caller is superseding instead
of inferring them from the complement of the replacement. A caller
computes a replacement from a history it read; anything that arrives
afterwards is not part of that, and is now left alone. IDs in both lists
are still kept, so read-modify-write callers are unaffected.

The test fake's ReplaceMessages was a slice assignment, which could not
observe ordering or identity at all -- it now mirrors the real stores.

Refs archie-core-6e0
2026-08-04 22:21:15 +10:00
sam d5e016ec59 fix(gateway): resolve the most recently active session, not an arbitrary one
resolve's comment claimed to find "the most recent session matching this
thread and bot" and took whichever the store returned first. On the
NellDB store that is document order -- by session ID -- so it was
effectively arbitrary. The tracker cache is in-memory, so every restart
re-resolves from the store, and the operator could be dropped back into
a session they had left with /new, answering against pre-/new history.
Probed over 10 restarts, 4 landed in the older session. LastActiveAt was
populated on every record and never read.

resolve now picks by LastActiveAt, which also makes it independent of
the two backends disagreeing: SQLite's GetByChannel orders created_at
DESC and NellDB's did not, and the conformance suite did not pin it.

listAllLocked reversed document order and passed that off as recency,
so List -- documented newest-first, and what /sessions and the dashboard
render -- returned reverse-lexicographic order. It now sorts on
LastActiveAt, falling back to CreatedAt for records never touched.

Refs archie-core-7s3
2026-08-04 22:13:55 +10:00
sam 0814f7b9b4 fix(gateway): give messages durable identity and a SQLite store
Three defects shared one root cause -- messages had no identity of their
own, so the store could not tell a redelivery from a new message:

- SaveMessages destroyed history. Writing a batch replaced records that
  a concurrent write had already stored, and earlier messages vanished.
- A redelivered channel message overwrote the stored one, contradicting
  the append-only sequence in docs/architecture/messaging-and-work-intake
  .md:112-114.
- /compress stranded whatever exceeded one page: deleting 1200 reported
  1000 deleted and left 200 behind.

Messages now carry an application-generated MessageID, derived from
(session, source id) so a redelivery lands on the same record, and
stored records are immutable -- a repeat write is a no-op rather than an
overwrite. SourceID keeps the channel-native id for correlation without
being load-bearing. _ts is written as Unix milliseconds and clamped
monotonic per session: Telegram dates are whole seconds, so replies
arriving inside the same second would otherwise order arbitrarily.
Deletes page with a cursor instead of assuming one page holds everything.

Adds a SQLite session store behind the same interface, per
docs/architecture/messaging-and-work-intake.md:107. A shared conformance
suite runs all 19 groups against both implementations, so the two cannot
drift while the NellDB path is still the one wired up.
2026-08-04 01:44:26 +10:00
sam bb77f88f73 fix: message history ordering and paged search
- Add Message.At timestamp as the ordering key, stamped at save when zero
- Store each session's messages in its own collection with time-ordered IDs
- Fix bulk saves overwriting history and sequence reuse after deletes
- Walk past NellDB's internal scan limit for history and deletes
- Add paged SearchMessages (MessageQuery/MessagePage) with truncation reporting
- Drop a session's message collection when the session is deleted
2026-08-03 19:08:40 +10:00
sam 712388dc6c fix: Phase 4 structural refactor — reduce lint findings from 28 to 5
cyclop (5→0): extract subcommand/case bodies in codesearch,
subprocess, Telegram, indexing
funlen (5→3): extract memory action handlers, tool provider startup
gocognit (7→2): extract config finalize/loadDirInto helpers, release
announcer, capture tool validation, Telegram gateway (200→108),
archie-agent main loop (35→0)
nestif (8→0): extract identity validation, container validation,
conf.d scanning, email multipart parsing, Telegram model/update
parsing, build-no-changes closure
interfacebloat (3→0): split Forge/IssueForge/PRForge/RepoForge,
SessionStore/SessionLifecycle/MessageHistory,
TaskStore/TaskLifecycle/TaskQueries/TaskEvents

Remaining: daemon process, workflow Feasibility/TDD/AgentStage,
archied main.go run (108→30 remaining). These require deeper
refactoring best done per-function in follow-up beads.

Updated contract tests and archied wiring tests to check both
main.go and the new telegram_setup.go.
2026-07-29 10:15:22 +10:00
sam a66da97ab7 gateway: implement session lifecycle and context commands (archie-core-alm.5)
Add gateway/runtime-owned commands that remain usable when LLM credits
are exhausted:

- /start — acknowledgement without consuming a reply
- /new [name] (alias: /reset) — create a fresh session
- /topic [off|help|<session-id>] — session switching for threaded chats
- /retry — remove last assistant message, replay last user turn
- /undo [N] — remove last N messages from history (default: 1)
- /title [name] — set or display session title
- /branch [name] (alias: /fork) — create child session with inherited history
- /compress [here N|focus topic|--preview|--dry-run] (alias: /compact)

Design:
- sessionTracker maps channel+thread to active session, auto-creating
  implicit sessions with the legacy sessionKey format for backward
  compatibility with existing persisted data
- /new and /branch create explicit UUID-backed sessions
- SessionStore extended with DeleteRecentMessages, MessageCount, and
  SaveMessages for undo/retry/branch/compress operations
- SessionContext extended with Title, ParentSessionID, BranchName
- All commands report 'not configured' when session store isn't wired

Session resolution is wired into the LLM responder so /new, /branch,
and /topic switches affect subsequent chat turns.
2026-07-29 03:43:34 +10:00
sam 15d746c2f6 style: phase 1 of lint remediation — autofix pass (qa4)
golangci-lint run --fix plus the formatters: errorlint, gocritic,
sloglint, usestdlibvars, usetesting, gci, gofumpt, goimports.
160 issues -> 130.

One autofix was wrong and had to be reverted by hand. gocritic rewrote

    ctx, cancel = context.WithTimeout(ctx, m.prefetchTimeout)

to `:=` inside an if-block in internal/memory/manager.go. That does not
compile, which is the good outcome — had it compiled, the shadowed ctx
would have scoped the timeout to the if-block and the prefetch deadline
would silently never have applied. Restored the assignment and commented
why it must not be a declaration.

Worth recording: --fix is not a safe no-op. It broke the build on a
24-file change, and the failure mode it was one compiler error away from
was a silent behaviour change.
2026-07-28 01:37:20 +10:00
sam f73b2d1c8c feat: memory manager sync/prefetch/scan/shutdown + broad in-flight epic work
Checkpoint commit covering concurrent work across several archie-core-abg
epics to avoid losing in-progress state:

- Memory manager (18.7-18.16): background sync pipeline, prefetch with
  timeout, memory-write notification, threat scanning, shutdown drain
  with abandoned-write tracking, frozen snapshot pattern. Wired
  ScanContent and NotifyMemoryToolWrite into HandleToolCall so scanning
  and notification actually apply to writes, not just exist as unused
  primitives.
- Fixed a literal unresolved merge-conflict that had been committed to
  internal/memory/scrubber/scrubber.go.
- Tool provider/registry, MCP client/transport (stdio, HTTP, SSE),
  multimodal results, disclosure.
- Pairing device auth store, rate limiting.
- Gateway compression, session store, tasks.
- Various workflow, plugin host, and agentexec updates.
2026-07-27 03:34:46 +10:00
sam b95993c44a fix: address adversarial review findings (21v + 19v)
Session persistence (21v):
- Mutex coverage on all nellSessionStore read operations
- Router.Store nil guard in handleStatus()

Config (19v):
- Remove dead code no-op in finalize() (Label='' assignment)
- Nil guard on loadFeatureFile() cfg parameter

40/40 packages pass with -race.
2026-07-26 13:56:25 +10:00
sam 948efde036 feat: session persistence — message storage, history retrieval, search
SessionStore extended with SaveMessage, RecentMessages, SearchMessages.
NellDB-backed implementation uses a separate DocDB collection for messages
keyed by session:seq. Conversation history loaded into LLM context for each
Telegram message. Responses persisted. Substring search fallback (vector
search deferred to memory provider embedding integration).

Adapter.Store() exported to share the underlying NellDB engine for session
collections without opening a second log.

Telegram LLMResponder now:
- Saves inbound messages to session store
- Loads 20 most recent messages as chat.Message history
- Sends full conversation context to llm.Chat()
- Persists the assistant response

All 38 packages pass with -race.
2026-07-26 10:33:13 +10:00
sam 0314955587 feat: wire memory, guardrails, and tool registry into daemon agent loop
Memory:
- MemoryManager created in main.go with built-in file-backed provider
- SystemPromptBlock() injected into every agent stage request via ExtraRules
- memoryPrompt() helper on Daemon, threaded through TaskContext.SystemPrompt

Guardrails:
- GuardrailEngine created from DefaultGuardrailConfig in main.go
- RecordSuccess/RecordFailure called after every agent stage run
- Threaded through TaskContext.Guardrails for per-task guardrail enforcement

Tool registry:
- NewRegistry() created at startup, MCP servers started from config
- Passed to Daemon.ToolRegistry for dynamic tool discovery

Daemon changes:
- Memory, Guardrails, ToolRegistry fields added (nil-safe, backward compat)
- TaskContext gains SystemPrompt + Guardrails fields
- Agent stage injects memory context into ExtraRules
- Agent stage records guardrail state after each run

Telegram LLM routing deferred (TODO comment) — gateway commands work.
All 38 packages pass with -race.
2026-07-26 10:25:07 +10:00
sam bc1caf67aa feat: BasePlatformAdapter interface + Session types (16.3, 16.5, 16.9, 16.10, 16.11)
adapter.go — SendResult struct (success, message_id, error, retryable,
retry_after, continuation_message_ids, error_kind), ChatInfo struct
(name, type, chat_id, thread_id), BasePlatformAdapter interface
(Connect, Disconnect, Send, ChatInfo).

session.go — SessionSource (platform, chat_id/name/type, user_id/name,
thread_id, guild_id, scope_id, profile, is_bot, metadata) with Key()
for deterministic session key. SessionContext wrapping source with
connected_platforms, home_channels, session_key/id, timestamps.

session_store.go — SessionStore interface (GetOrCreate, Get, Reset,
List, GetByChannel, Touch, Delete, Close) + nellSessionStore backed
by nell.DocDB with mutex-guarded Save/Delete for race safety.

Verification: 4 findings fixed (gofmt alignment, race on mutex,
!= true anti-pattern, missing zero-value assertions). 58 tests pass
(incl -race), go vet + gofmt + golangci-lint clean.

Closes: archie-core-abg.16.3, 16.5, 16.9, 16.10, 16.11
2026-07-26 01:29:36 +10:00