Every install serves from Postgres, so the legacyread/legacyimport packages,
the archie-state-store import command, the boot gate that refused to serve
over unimported SQLite files, and db_path (which only located those files)
go. Migration 0010 drops the import_completion table.
Each credential is now only a {engine, key} secret reference. A stale
token_env in a file is reported at startup as an unrecognised key.
nats.token_env stays (no secret-ref form), and providers.api_key_env stays:
it is the carrier the resolver writes each resolved key's env name into.
None had a production consumer: [agent] and containers.enabled were decoded
only so old files kept loading, [oidc] was never read (archie-ui takes OIDC
from flags), and [image] was validated but no provider exists.
The embedded NATS store and endpoint file, the task-log registry and the
readiness disk probe resolve from state_dir (default: the data home's archie
directory) instead of the directory of db_path. db_path now only locates the
legacy SQLite files for the one-time import and its boot gate.
The index built google/codesearch over the chat workspace on every daemon
start, for the process lifetime, through a child-process helper and a
SQLite metadata DB -- and grep consulted it only for searches confined to
the workspace root, which the analysed sessions showed was a small
fraction of searches. The compute was continuous; the benefit was not.
Removes internal/indexing, the workspace-codesearch helper subcommand,
the [indexing] config section, and the GrepIndex accelerator seam from
the grep tool, which returns to its direct walk. google/codesearch
leaves go.mod with it.
Review of the two previous commits found the split was incomplete in both
directions: the checks moved off the file load path were still enforced
anywhere the value could enter the system, and the fold reached only one of the
three overlay paths.
A stale database-owned TOML value could still stop archie-state-store. The
process seeds every control-plane resource from the same file config on a fresh
database, and the seed runs the resource's own validator; three of the five
values the loader now accepts aborted the boot there (verified by booting
RunStateStore before the fix). A seed that does not validate is now SKIPPED and
reported instead of fatal: the resource stays absent, so the file's value is the
one in effect and editing config.toml is still the fix -- the daemon and the
Gateway are where it fails closed, at boot.runtimeConfig.
The resource validators were weaker than configuration.Validate for exactly the
fields the split reclassified, so a value the file path used to refuse at load
became storable in the database, and then un-fixable by editing config.toml.
The trigger enum and the test-glob rule now come from the configuration package
rather than being restated, the container image is required there as it is in the
file, and a table test ties the two layers together value by value.
The directory form of -config-overlay kept the archie-core-e2e2 failure: it
decoded each overlay file straight into the populated config, so a map-valued
entry it only partly addressed still cleared the fields it did not name. Both
the overlay layer's main file and its feature files now fold, which is what
Loader.Dir's documented parity with Loader.Overlay has always claimed.
The overlay file's key report is derived from the apply's own view of the file as
well as the file decode's, because TOML matches keys case-insensitively and the
fold's yaml decode does not: a key only the file decode matched was consumed by
the report and dropped by the apply, a silent no-op that worked in the base file.
Tests: the state-store boot is asserted against the real process, the directory
overlay and the case-mismatched key each have one, and ImportConfig pins that a
refused seed is not stored.
Loader.finalize ran the whole validation over the file document, so a stale
TOML value in a setting the control plane owns failed startup -- including
archie-state-store, which resolves its file config and opens the store without
ever layering a database resource over it. docs/prds/runtime-control-plane.md,
"Bootstrap, migration, and recovery": after migration, settings in TOML are
ignored and cannot block State Store startup.
The checks over the settings a control plane resource is seeded from and
replaces wholesale now run only on the effective document: providers
(provider-settings), poll_interval and dispatch (scheduling-policy), containers
(container-runtime-policies) and the repository lists (repository-policies).
They moved from validate to validateDatabaseOwned; validateBootstrap keeps the
checks over settings no resource carries (forge intake, identities, nats,
memory, image, capture). File sources pass validateBootstrap and
Loader.ApplyOverlay, which layers overrides onto an effective document, passes
Validate.
Nothing stops being judged: the daemon and the Gateway both run the full
Validate in boot.runtimeConfig, the one place database resources are layered
in, before any capability starts -- so a stored value that will not validate
still stops the process rather than being published. Cases that asserted the
load path rejects a database-owned value now assert both layers, and
TestRuntimeConfigFailsClosedOnAnInvalidDatabaseValue pins the second.
Loader.overlayFile decoded the overlay file straight into the config the
base file produced, and a decode replaces a map-valued entry wholesale:
[services.state] target = ... in an overlay cleared the target_token the
base file set. That is archie-core-e2e2 on the user-visible -config-overlay
path -- deployments/dev.toml is the documented argument and sets
[services.state] target/listen, while target_token lives in the installed
config.toml. An enabled image.hosted entry lost its class and api_key_env
the same way, silently turning the provider off.
apply.go's foldOverrides is the fix that path was missing, so the file
overlay now goes through the same fold as the map overlay: the overlay file
parses to its raw mapping (decodeFileMapping), the fields of each partially
addressed map entry are folded forward out of the base config, and the
completed mapping is decoded over it. Loader.ApplyOverlay, ApplyOverlayValues
and apply.go stay: the hazard they were written for is this one.
An overlay file still reports its own unknown keys, from a separate decode
into a scratch value.
Autonomous workflows now always hand a complete task to a scoped archie-agent container. Embedded NATS is the default and external NATS changes only broker placement; legacy agent and containers.enabled inputs decode for migration but cannot select an execution path. The daemon flushes task-event subscriptions before a worker can publish, and runtime updates snapshot and restore the configured worker image.
Autonomous workflows now have exactly one production boundary: a task-scoped archie-agent receives a full-task core-NATS request and runs the worker-local stage loop. The legacy [agent] section remains decode-only for migration; it no longer selects runtime behavior.
Render completed ToolCallEvents as Telegram-safe streaming blocks
with compact name/status and a short cleaned preview. Summarize
file contents and listings instead of dumping source or quoted
JSON. Collapse repeated legacy turn-budget failures.
Remove max_tokens turn gating and the 200000 turn_budget_chars
default so absent or commented config cannot restore an aggregate
cap. Keep per-result max_result_chars and spill.
Confine find/grep discovery to the active workspace/worktree.
Refs #604
Task and event persistence now uses a dedicated SQLite sidecar derived from the configured state prefix, matching the independently persisted conversation store. The configured path now means a state-path prefix rather than a database file so fresh SQLite state never attempts to interpret the previous storage format.
Remove the retired implementation and dependency completely, strengthen SQLite migrations and concurrency behavior, restore backend-neutral coverage, and align configuration, operational guidance, UI labels, and deployment documentation with the new storage and host-daemon topology.
Loader.Resolve() selects TOML/YAML files or a config directory as the
production entry point, replacing the TOML-only Overlay() as the default
path. Add internal/channels.Manager, reporting each adapter's actual
runtime lifecycle (configured/starting/running/degraded/failed) instead of
inferring health from configuration presence alone. Wire channel reload
through a per-adapter seam.
Fixes#55
internal/infrastructure/configuration/setup.Run drives operator
identity, LLM provider, models, forge, and Telegram chat channel
prompts and returns TOML edits -- not a config.Config. Building a
parallel in-memory representation to validate against was rejected
during design review: it's a second source of truth about the exact
thing (text vs. struct disagreeing) this whole feature exists to stop.
Tests instead render the edits against the real embedded template and
load the result through a real configuration.Loader, proving the
actual defaults/decode/validate path accepts it.
Two boot-time landmines in config.example.toml surfaced by that
approach, both fixed here:
- [providers.openai] ships active with api_key pointed at the bws
engine. cmd/archied/provider_secrets.go resolves every entry in
cfg.Providers at startup, not just the ones [models] references, and
refuses to start the daemon if any fails -- so picking any other
provider (or picking openai but skipping the key) left a config that
wouldn't boot on a machine without the bws CLI installed (the common
case). stepProvider now neutralises providers.openai.api_key to the
zero-value secret ref whenever openai wasn't the actual choice.
- [[repos]] shipped active with a placeholder owner/name that
validates cleanly (owner/name are only checked non-empty) and would
have archied silently polling a repo that doesn't exist. Commented
out in the template with an explanation; setup does not write this
section since tomlwrite cannot target array-of-tables.
TestExampleConfigLoads updated to expect zero repos.
Also: forge and provider secret env-var names are each defined in one
table (forgeToken, cloudProviders) read by both the config edit and
the secret sink, so the two can't name a secret differently -- the
exact bug class this feature exists to close for the installer, tested
directly (TestRun_ForgeTokenKeyMatchesStoredSecretKey compares the
decoded key against the key the secret was actually stored under, not
a hardcoded literal on both sides). Telegram writes chat.telegram.
token_env, not a secret.SecretRef -- TelegramConfig has no such field.
Telegram also requires at least one allowed_user_id: AllowedUserIDs is
deny-by-default, so a bot token with none configured answers nobody,
which reads as broken rather than as the safe default it is.
SecretSink is a caller-supplied sink (Put buffers, Commit persists),
not a return value: Run never hands back plaintext secrets, and the
caller is expected to call Commit only after proving the generated
config loads -- so a validation failure never leaves a secret written
without the config that references it.
Refs archie-core-cbk
Test fixtures used 'sam', 'samcharles93', and 'sam/<repo>' as owner
names. These are personal and machine-specific — the repo should use
generic placeholders like 'acme' (matching the existing convention in
worktree_test.go).
config.example.toml: samcharles93 -> your-github-username
Test owner: sam -> acme
Repo refs: sam/archie-core -> acme/archie-core, etc.
Example skills: sam -> acme
These asserted on file contents rather than what the code does:
Dockerfile strings, deploy.yml YAML, CHANGELOG headings, and greps of
main.go for identifiers. Two also referenced config.docker.toml and
docker-compose.yml, which were never committed, so they could not pass
at all.
The worst compared strings.Index byte offsets in main.go to assert
startup ordering -- it would pass on code that never runs and fail on
a correct refactor that moved a line.
internal/config both defined the types every package imports and performed
the file I/O to produce them. Importing a settings struct therefore pulled
in a TOML decoder and a YAML decoder, and the shared model owned discovery,
decoding, overlays, defaulting and validation all at once. See
docs/architecture/configuration.md, which requires loading to move out and
internal/config to be dissolved.
internal/config is now types only -- no os, no filepath, no toml. The new
package owns loading, one concern per file:
features.go the closed set of recognised feature files
discovery.go directory scan and classification
decode.go TOML/YAML decoding, format dispatch
defaults.go filling absent values, no errors
validate.go judging supplied values, no mutation
provenance.go which file contributed what
loader.go Loader wiring the above; File/Overlay/Dir
Beyond relocating, this fixes what the old code conflated:
- Defaulting and validation were interleaved. applyAgentDefaults and
applyDispatchDefaults returned validation errors; validateContainers and
validateSingleIdentity quietly mutated the config. There was no point at
which you could see what the operator actually supplied. They are now
separate passes: applyDefaults never fails, validate never writes.
- Errors were fmt.Errorf strings prefixed "config: ", so a caller could not
distinguish operator-fixable input from an unreadable file without
matching message text. Five sentinels now carry that distinction.
- dataHome() read XDG_DATA_HOME directly and discarded the error from
os.UserHomeDir, so defaulting could not be tested without the
environment. It is injectable on Loader and no longer swallows the error.
- Nothing recorded where a value came from. With overlays, feature files
and conf.d/ all able to set the same field, "which file set this?" was
answerable only by re-deriving precedence by hand. Provenance records it.
- Feature files were applied in map iteration order, making load order and
any resulting error arbitrary. Discovery now returns them sorted.
- loadMainConfig reported "no config.yaml or config.toml found in %s" with
an empty string: the directory was lost when it was extracted from
loadDirInto, so the error named no directory.
- Unreadable or malformed conf.d/ extras were skipped silently, which
presents as a setting the operator wrote simply not taking effect.
- Decode errors from the main config were returned bare, without naming
the file that failed to parse.
The 40 loader tests moved with the code and their assertions are unchanged,
so the defaulting and validation rewrite is behaviour-preserving.
Document.Config is the temporary seam: still internal/config.Config, named
as a field so the compromise is visible at each use site, with its deletion
condition recorded in doc.go. Building typed per-domain sections now, while
every consumer still reads config.Config, would mean two shapes kept in sync
by hand with no consumer for the new one; that belongs with the dissolution
work, not ahead of it.
No consumer outside cmd/archied changed.
Verified: go build, go vet, gofmt clean; go test ./internal/config/... and
./internal/infrastructure/configuration/... pass. Pre-existing gateway and
telegram failures are unchanged.