Commit Graph
26 Commits
Author SHA1 Message Date
sam 3a965e2a4f feat(nats): an external broker defers logins to archie through auth callout (#1866)
Task containers log in with their run credential on an external broker and
reach only their own subjects, decided by the embedded broker's authenticator.
The broker token is never put in a container environment.
2026-11-14 04:04:15 +11:00
sam c54d328f0d feat(store): run services as a non-superuser role and name every connection (#1857)
Every service connected to PostgreSQL as the database owner, which is a
superuser, so anything that compromised an app process owned the cluster, and
every connection reached the server with an empty application_name, so
pg_stat_activity could not say which service a session belonged to.

Split the DB credential into two login roles and name every pool:

- archie_migrator owns public and its objects and runs goose. Migrations run
  on a short-lived pool opened from the new migration_database_url, before the
  serving pool opens. It is not a superuser.
- archie_runtime serves requests with SELECT, INSERT, UPDATE, DELETE, sequence
  USAGE/SELECT and function EXECUTE only. It has no CREATE privilege, so it
  cannot change the schema.
- Every pool sets application_name: archie-state-store, archie-gateway, and a
  -migrate / -recovery / -schema / -provision suffix on the transient
  administrative connections. Existing one-URL installs keep working: an
  absent migration_database_url falls back to database_url and the migration
  runs on the serving pool exactly as before.

Provisioning is a new idempotent `archie-state-store provision` subcommand: it
needs an admin URL the services never hold (database owner or superuser),
creates or reconciles both roles, hands every public-schema object to the
migration role, revokes PUBLIC's CREATE on public, sets row-level grants, and
sets the migration role's default privileges so tables future migrations
create are usable without a hand-run grant. It stores SCRAM verifiers, refuses
superuser roles, and verifies the boundary before committing.

install.sh now provisions the bundled path and separates the credentials: the
superuser password the Compose database is created with lives only in a
compose-only postgres.env, while the env file the units load holds the
non-superuser PGPASSWORD. A bundled install that predates the split is
upgraded in place: the roles are created, the superuser password is rotated
out of the env file, and config.toml is moved to the runtime and migration
URLs. External databases keep their operator-managed credential; the
deployments README documents the one-shot provision command.

setup writes both bundled URLs, dev:postgres provisions the dev database, the
e2e stack provisions its throwaway server, and the deployment templates point
at the two roles.
2026-11-11 03:12:54 +11:00
sam 3b6de96782 docs(deploy): name the registry each agent-image path publishes to
The pinning section said CI republished ghcr.io on every push to main. CI
publishes git.catlow.cloud, and only for a release tag; ghcr is where the
hand-run tools/publish.sh writes, which is the reference the default names.

A release zip pins a digest on a registry an anonymous pull cannot reach, and
[containers].registry_auth -- the setting that fixes it -- was documented
nowhere. Documented there, beside the pin.

Also corrected: every shipped profile does not pin the image, and the section
link omitted its number so it matched no heading.
2026-10-31 00:35:49 +11:00
sam 66daa071a5 fix(configuration): raise the default max uptime above the task limit 2026-10-26 18:32:25 +11:00
sam 3b7193c207 refactor!: remove the diff-size cap and its gate.diff-size step 2026-10-24 22:19:23 +11:00
sam 9cdae5642e feat(rpc): mTLS between services from operator-issued certificates
Replace the service bearer tokens with [tls] ca_file and cert_dir. Caller
identity comes from the verified certificate, not metadata. Without TLS,
services listen and dial on loopback only. Task containers get only the CA
and keep their grant.
2026-10-16 02:37:51 +11:00
sam 209f2b4adb docs: server quickstart, standalone template first, install starts PostgreSQL 2026-10-06 01:43:16 +11:00
sam c73ff313ea refactor(config): drop unused dispatch state labels
The daemon stopped setting forge state labels when label-driven retry was removed; the config field, helpers and defaults had no consumer. Remove them and the Scheduling policy page's State labels section.
2026-10-05 15:05:06 +11:00
sam 75988204d5 docs: strip doc and issue references from config, deployments and examples 2026-10-03 22:27:08 +10:00
sam c1cd5c8f5b chore(state-store): remove the one-time SQLite import and db_path
Every install serves from Postgres, so the legacyread/legacyimport packages,
the archie-state-store import command, the boot gate that refused to serve
over unimported SQLite files, and db_path (which only located those files)
go. Migration 0010 drops the import_completion table.
2026-09-25 00:40:48 +10:00
sam e5105280ad chore(config): remove token_env aliases for forge, artifacts and telegram
Each credential is now only a {engine, key} secret reference. A stale
token_env in a file is reported at startup as an unrecognised key.
nats.token_env stays (no secret-ref form), and providers.api_key_env stays:
it is the carrier the resolver writes each resolved key's env name into.
2026-09-24 23:58:26 +10:00
sam 9864766804 docs: describe the PostgreSQL-only stores, state_dir and the import-only db_path
Architecture, deployment, installer, skill and PRD text now name the Postgres
State Store, the offline recovery commands on database_url, state_dir for host
state, and db_path as the legacy-file locator for archie-state-store import.
Finalised PRDs, CHANGELOG, docs/news and docs/archive are unchanged records.
2026-09-24 14:51:48 +10:00
sam feeaa16c7b feat(postgres): wire the State Store to Postgres — DB URL, fail-closed pool, composition seam
Wave 1 of the Postgres migration (archie-core-7r0x.9). This is the shared shape
every port lane builds on; nothing here switches a live path to Postgres yet.

- config: add `database_url` beside `db_path` (config.go), mark it bootstrap-only
  in denied_keys, expose it in the webui StorageView/config schema, and add it to
  config.example.toml plus all deployments/*.toml.
- docker-compose: add a postgres:18 service; `task dev` (dev:postgres) brings it
  up and waits for it before the State Store boots.
- archied: `openStateStore` now opens and migrates the process-scoped Postgres
  pool via `openStateStorePool`. The State Store fails closed (D1.2) when
  database_url is absent or invalid rather than fall back to SQLite; serving
  against Postgres later requires an import-completion record (D1.3), which is a
  later lane's gate, not this one.
- composition seam: task-store construction (state_store_task.go) and EDA-store
  construction (state_store_eda.go) are extracted from openStateStore, with the
  pool open in state_store_pool.go. This is a behaviour-preserving refactor only
  — no store, interface, error, ordering or logging change — so L1 and L2 build
  against separate files instead of sharing state_store.go.

Tests: fail-closed pool behaviour is pinned (absent/invalid URL); the State Store
boot/restart tests now run against a throwaway Postgres 18 container (pgtest
TestMain). Scoped test command and module-wide build pass; see lane report.
2026-09-24 02:18:56 +10:00
sam 851c46d8ed docs(deploy): UI Service deployment, auth, and operations profiles
Bead archie-core-8cda.5.5:

- task dev now runs archie-ui (dev:dashboard) on :8484 -- after the
  cutover archied binds no dashboard listener, and Vite's /api,/events
  proxy needs a live receiver there. dev:mobile starts archie-ui with a
  LAN bind and -token-file (reusing the pre-cutover token path) instead
  of a mobile archied overlay; dev:ui's summary points at archie-ui.
- deployments/dev.toml documents itself as the archie-ui overlay and
  states the [services.gateway] target explicitly.
- systemd-user-service.md: the auth gap is now the documented posture --
  archie-ui has no secret registry, resolves GATEWAY_TOKEN/
  STATE_STORE_TOKEN from its process environment only, so the starter
  (EnvironmentFile) exports them; worked env-file example added, plus
  /healthz as the token-free liveness probe.
- INSTRUCTIONS.md ships five binaries with archie-ui in the quickstart
  and the token story; README.md carries the auth note; the
  docker-nats-stack profile notes archie-ui binds [web] and dials the
  same [services.*] targets.
- release tooling: archied's component closure extends with
  cmd/archie-ui and internal/app/archieui, so a UI-process-only commit
  can no longer fall between the two changelogs; RELEASING.md documents
  that the UI Service ships inside the archied release.
2026-09-10 01:14:14 +10:00
sam 05529d48d5 refactor(state-store): cut daemon/Gateway over to the remote State Store (archie-core-8cda.4.6) (#774)
* refactor(state-store): delete NATS storerpc; agent store is gRPC-only (archie-core-8cda.4.6)

The agent is fully on the gRPC State Store contract, so the legacy NATS
storerpc path is dead (docs/prds/state-store-contract.md §12 step 4):
- delete internal/storerpc (Server, Client, subjects) and its tests
- drop the storerpc registration from registerTaskRPCServers and the
  now-unused store param
- make the agent transport's State Store URL required and Store() return
  only the gRPC staterpc.Client (no NATS fallback)
- rework agentworker/agent transport tests to back the store with a real
  gRPC State Store server instead of the deleted NATS storerpc server

The agent binary now fails fast when STATE_STORE_URL is absent. Agent store
calls (workflow.Store) go over gRPC to the archie-state-store service.

* refactor(state-store): delete daemon/gateway in-process store ownership (archie-core-8cda.4.6)

The daemon and gateway no longer open archie.db in-process: the single SQLite
file is owned by the standalone archie-state-store process, and both consumers
dial its gRPC State StateStore contract (docs/prds/state-store-contract.md
§12 step 7, no dual-store ownership).

- openStores no longer opens the production task store or seeds b.st; it only
  wires secrets/forge and the gateway conversation store
- openStateStoreAdapter now requires [services.state].target (the old local
  default is deleted) and resolves b.stateStore as the remote *staterpc.Client
- delete the in-process startStateStoreServer (the multiplexed gRPC server and
  its per-task bearer-token registry) from the daemon composition
- RunGateway now resolves the same remote State Store adapter; its session
  SQLite stays Gateway-owned and untouched
- daemon.ConnectedStateStore carries the configured target plus the resolved
  bearer token, which containerEnv injects as STATE_STORE_URL / STATE_STORE_TOKEN
  so archie-agent authenticates to the same remote service
- readiness store probe reads b.stateStore

PRD-consistent choice (noted): with the in-process server gone, per-task token
generation is removed; the daemon/gateway/agent all present the single
[services.state].target_token (or STATE_STORE_TOKEN secret) that the standalone
store validates.

* docs(webui): require [services.state].target in config and deployment templates (archie-core-8cda.4.6)

After the in-process store path is deleted, archied and archie-gateway dial the
standalone archie-state-store gRPC service via [services.state].target and never
open archie.db themselves (docs/prds/state-store-contract.md §12 step 7).

- config.example.toml documents [services.state].target as required, with the
  loopback vs Docker-bridge gateway + target_token rule
- deployment templates set [services.state] per topology: bridge gateway +
  token for container-mode agents, loopback for host-only
- deployments/README.md and systemd-user-service.md describe start commands
  for both topologies and drop the stale "not yet consumed by the daemon" note

* style: gofumpt alignment for State Store daemon literal; appease noctx/staticcheck (archie-core-8cda.4.6)

* docs(state-store): refresh stale in-process references post-cutover (archie-core-8cda.4.6)
2026-09-07 05:06:12 +10:00
sam 5bd553d8ed Merge branch 'feat/gateway-service': Phase 1 Gateway Service extraction
Stands up archie-gateway as its own binary serving ChatContract over gRPC
and makes it the daemon's only transport for webui chat (Q3 part 5's
atomic flip -- the in-process serving path is deleted, not deprecated).

- contract seams + protobuf toolchain (Phase 0)
- single taskactions.Service across webui, daemon NATS responder and the
  standalone gateway's client
- services.gateway.target is the only config surface left; mode is gone
  now that remote is the only transport
- task dev launches archie-gateway alongside archied + Vite

Verified end-to-end against live infra: real GitHub token (zero repos
configured, so no repo was polled or touched) and the real
@heyarchie_home_bot Telegram bot, both processes booted together against
shared NATS/store, chat and /status round-tripped successfully.

# Conflicts:
#	go.sum
2026-09-06 02:26:19 +10:00
sam 14e5b542bf fix(gateway): drop services.gateway.mode, remote is the only transport
Mode had exactly one legal value after the previous commit deleted the
in-process path, which made it dead config surface that could only
confuse an operator into thinking there was still a real choice.
ServiceConnection is now just a target address; composeChatContract only
checks it's non-blank, defaulting to 127.0.0.1:8585.
2026-09-06 02:13:03 +10:00
sam 0a6496d2cd feat(gateway): extract ChatContract service 2026-09-05 23:06:16 +10:00
sam 2d8a610bab fix(deploy): pin archie-agent image away from :latest (#751)
ghcr.io/samcharles93/archie-agent:latest is re-published on every main
push that touches the runtime's package closure, so a bare :latest
reference can change under a running deployment with no warning and no
way to reproduce a prior build.

archie-agent has no published semver tag yet -- CI (deploy.yml) only
ever pushes :latest to GHCR, confirmed via the GHCR package API. Pin
docker-compose.yml and the GitHub deployment profiles to the verified
digest of :latest as of 2026-09-05
(sha256:870f1b1357979cfea23fccacb24166ebeb9ab321708ef2c2b218a763ed99b090),
and document the pinning strategy -- including the semver-tag-once-
published preference and digest pinning for production -- in
deployments/systemd-user-service.md. Update the generic
docker-nats-stack.toml example to demonstrate the same convention.

Ref archie-core-kllw (GitHub #670)
2026-09-05 22:19:05 +10:00
sam 37cfe87508 fix(webui): validate mutation Origin against the effective external scheme
Origin validation compared the request's Origin against the daemon's own
direct connection: scheme from r.TLS, host from r.Host. Behind a TLS-
terminating reverse proxy the browser sends Origin: https://host while the
daemon sees a plain HTTP connection, so every dashboard mutation was
refused as cross-origin.

Taking X-Forwarded-Proto/-Host unconditionally would be worse than the bug:
any client can set them, so an exposed daemon would let an attacker forge
the scheme half of the CSRF check. They are honoured only when
web.trust_forwarded_headers is explicitly enabled, which is off by default
and documented as safe only behind a proxy that overwrites or strips
untrusted forwarded headers.

authorizeTaskMutation becomes a method because the decision now depends on
server configuration; its three call sites move with it.
2026-08-25 02:06:02 +10:00
sam 972fb5b8e1 feat(deployment): default to managed NATS workers
Autonomous workflows now always hand a complete task to a scoped archie-agent container. Embedded NATS is the default and external NATS changes only broker placement; legacy agent and containers.enabled inputs decode for migration but cannot select an execution path. The daemon flushes task-event subscriptions before a worker can publish, and runtime updates snapshot and restore the configured worker image.
2026-08-24 14:56:57 +10:00
sam 691e10c251 fix(telegram): compact tool output and drop turn caps
Render completed ToolCallEvents as Telegram-safe streaming blocks
with compact name/status and a short cleaned preview. Summarize
file contents and listings instead of dumping source or quoted
JSON. Collapse repeated legacy turn-budget failures.

Remove max_tokens turn gating and the 200000 turn_budget_chars
default so absent or commented config cannot restore an aggregate
cap. Keep per-result max_result_chars and spill.

Confine find/grep discovery to the active workspace/worktree.

Refs #604
2026-08-23 16:40:16 +00:00
sam 61b4b7b0cf refactor(store): replace legacy persistence with SQLite
Task and event persistence now uses a dedicated SQLite sidecar derived from the configured state prefix, matching the independently persisted conversation store. The configured path now means a state-path prefix rather than a database file so fresh SQLite state never attempts to interpret the previous storage format.

Remove the retired implementation and dependency completely, strengthen SQLite migrations and concurrency behavior, restore backend-neutral coverage, and align configuration, operational guidance, UI labels, and deployment documentation with the new storage and host-daemon topology.
2026-08-09 22:27:25 +10:00
sam db8352744b fix(deploy): correct pull_policy in the docker-nats-stack template
'always' against a private registry answers 401 because the daemon
calls ImagePull with no credentials. The live config already uses
'missing'. This just brings the template in line.
2026-07-31 02:04:48 +10:00
sam fd1abdd6c8 fix(deploy): stop the agent stanza starting, and pull agent images
The agent service created a container that exited 0 immediately while
compose reported it as "Started", which reads like a crash loop. A build
profile keeps it out of `up -d`; naming it still works for build and pull.

containers.pull_policy was "if-not-present", which is not a recognised
value -- the pool pulls only when it is "always" or "missing", so the
typo disabled pulling outright and the deployed agent image drifted a day
behind the daemon. Watchtower cannot cover for it: agent containers are
spawned per task, and it only restarts long-running ones.
2026-07-31 01:51:19 +10:00
sam 3ee3bcf202 chore(deploy): commit the compose stack, drop the required overlay
docker-compose.yml was never committed -- it only existed on the
deployment host. It bind-mounted ./config.docker.toml and passed
-config-overlay unconditionally, which made an untracked repo-root
file load-bearing: Docker creates a directory when a bind source is
missing, and the loader rejects a directory, so a fresh clone took the
daemon down at boot.

The stack now passes a single -config pointing at a path outside the
repo. Nothing in the working tree is required at runtime. The
docker-specific values move to deployments/docker-nats-stack.toml as a
reference config, alongside the other deployment guides.

Also drops the ./:/workspace mount, which existed only for
desktop-commander's work_dir.
2026-07-31 01:13:23 +10:00