From 33751abe4e6db99f9889e984be9d019e78a5fce5 Mon Sep 17 00:00:00 2001 From: Sam Date: Mon, 7 Sep 2026 05:47:36 +1000 Subject: [PATCH] test(staterpc): verify State Store end-to-end and preserve context error identity (#775) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * test(state-store): real-process smoke + deadline/sentinel error contract Add end-to-end verification for archie-core-8cda.4.7, the .4.7 State Store verification stage against docs/prds/state-store-contract.md (rev. 2c): - Real-process smoke coverage: build ./cmd/archie-state-store, run it against a minimal config on loopback, poll the readiness surface, and drive the State Store gRPC contract through staterpc.NewClient exactly as a consumer does. Covers the combined task/event/capture view, error-sentinel fidelity across the wire, the found=false-not-error convention, single-owner SQLite (the binary owns the one archie.db file; the consumer never opens a store file), per-call deadline behaviour, restart/recovery (SIGTERM + reopen on the same db), and the remote capture/mapping/binding surfaces. - Fix a real deadline-identity gap in unmapError: a cancelled or deadline-exceeded context surfaced by gRPC-Go as a *status.Error was re-wrapped into a sanitised internal-style error, breaking errors.Is(err, context.Canceled/DeadlineExceeded). The agent's deadlineStore bounds each Store call with context.WithTimeout and the workflow consumer (workflow.go) uses errors.Is(err, context.DeadlineExceeded) to mark a stage interrupted rather than failed, so the re-wrap would wrongly park an interrupted task. Unmap Canceled/DeadlineExceeded codes back to the standard context sentinels. - Pin the error-mapping contract in staterpc: context-identity preservation, sentinel round-trip fidelity for every store sentinel, and the sanitisation rule (an Internal error must not leak raw detail or map to a sentinel). * test(state-store): assert single-owner SQLite composition (daemon/gateway require target) The single-owner invariant (docs/prds/state-store-contract.md §12 step 7/8) is that only archie-state-store opens archie.db; the daemon and gateway resolve their task-store surface from openStateStoreAdapter, which requires [services.state].target and opens no local store. This test pins that composition root so a future change cannot silently reintroduce a local *store.Store in the daemon or gateway path (dual-store ownership). * test(state-store): skip real-process smoke tests under -short The smoke tests build and exec the real archie-state-store binary, which is slow enough that a -short run (e.g. a focused edit loop) should skip them. task check (go test ./... -count=1) does not use -short, so they still run in the full gate. * test(state-store): fix subprocess log race under -race The smoke test captured the subprocess's stdout/stderr in a strings.Builder that the os/exec writer goroutine and the test reader goroutine touched concurrently, racing under -race. Replace it with a mutex-guarded buffer, the standard pattern for reading a running subprocess's output. * test(state-store): satisfy lint (noctx, errorlint) on new tests - exec.Command -> exec.CommandContext, http.Get -> NewRequestWithContext + Client.Do, so the subprocess and readiness polling honour the test context (noctx). - errorlint: compare the raw non-status error via errors.Is rather than != on a wrapped error. * docs(deployments): document State Store readiness verification and single-owner invariant The deployment runbook described how to launch archie-state-store but not how to verify it is healthy before starting archied. Document the -ready-addr readiness surface (/healthz, /health/detailed state_db component) and the single-owner invariant: one task SQLite file owned by archie-state-store, with archied/archie-gateway never opening archie.db directly (state-store-contract §12 step 7/8). * fix(state-store): preserve context deadline/cancel identity in mapError The symmetric server-side gap to the unmapError fix: if a store query fails because the caller's context expires mid-call (§6 StoreTimeout), the handler's mapError folded context.Canceled/DeadlineExceeded into codes.Internal. The client would receive a sanitised internal error with no deadline identity, so the agent's workflow consumer (errors.Is(err, context.DeadlineExceeded)) could not distinguish an interrupted stage from a failed one. Map a context cancel/deadline error to its matching gRPC code so unmapError rehydrates it back to the standard context sentinel, completing the §6/§7 round-trip. Pin both directions with TestMapErrorPreservesContextIdentity. --- cmd/archie-state-store/main_test.go | 478 ++++++++++++++++++ deployments/README.md | 19 + .../app/archied/state_store_client_test.go | 26 + .../infrastructure/staterpc/error_test.go | 127 +++++ internal/infrastructure/staterpc/values.go | 32 +- 5 files changed, 680 insertions(+), 2 deletions(-) create mode 100644 cmd/archie-state-store/main_test.go create mode 100644 internal/infrastructure/staterpc/error_test.go diff --git a/cmd/archie-state-store/main_test.go b/cmd/archie-state-store/main_test.go new file mode 100644 index 000000000..42137f3d7 --- /dev/null +++ b/cmd/archie-state-store/main_test.go @@ -0,0 +1,478 @@ +// Package main smoke-tests the standalone archie-state-store process by +// building the real binary, running it against a minimal config on the +// loopback interface, and driving it through the State Store gRPC contract +// exactly the way a consumer (archied, archie-gateway, archie-agent) does. +// +// This is the .4.7 end-to-end verification for +// docs/prds/state-store-contract.md (rev. 2c): it proves the process owns the +// single archie.db task SQLite, serves the combined task/event/capture +// surface over one gRPC service, preserves error-sentinel fidelity across the +// wire, honours a per-call deadline, survives a restart/recovery cycle, and +// is the sole owner of the SQLite file. See §5 (binary layout), §6 (handoff), +// §7 (error semantics), §11 (remote adapter) and §12 step 7/8 (single owner). + +package main + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "os" + "os/exec" + "path/filepath" + "runtime" + "strings" + "sync" + "syscall" + "testing" + "time" + + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" + + "github.com/samcharles93/archie-core/internal/domain/binding" + "github.com/samcharles93/archie-core/internal/domain/mapping" + "github.com/samcharles93/archie-core/internal/events" + "github.com/samcharles93/archie-core/internal/infrastructure/staterpc" + "github.com/samcharles93/archie-core/internal/store" +) + +// repoRoot walks up from the test's source directory to find the module root +// (where go.mod lives). The test runs in the package directory +// (cmd/archie-state-store), so the module root is above it; walking keeps the +// result correct regardless of the harness working directory. +func repoRoot(t *testing.T) string { + t.Helper() + _, file, _, ok := runtime.Caller(0) + if !ok { + t.Fatal("cannot determine test source path") + } + dir := filepath.Dir(file) + for { + if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil { + return dir + } + parent := filepath.Dir(dir) + if parent == dir { + t.Fatal("go.mod not found above test source") + } + dir = parent + } +} + +// buildBinary compiles ./cmd/archie-state-store into dir and returns the path. +// This is the real binary the smoke test exercises, not a test double -- the +// point of .4.7 is to verify the actual process. +func buildBinary(t *testing.T, dir string) string { + t.Helper() + bin := filepath.Join(dir, "archie-state-store") + cmd := exec.CommandContext(t.Context(), "go", "build", "-o", bin, "./cmd/archie-state-store") + cmd.Dir = repoRoot(t) + cmd.Env = os.Environ() + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("go build archie-state-store: %v\n%s", err, out) + } + if _, err := os.Stat(bin); err != nil { + t.Fatalf("built binary not found: %v", err) + } + return bin +} + +// writeMinimalConfig writes a config.toml that passes configuration.Validate +// while keeping the process minimal: the standalone binary reads its gRPC +// listen address from -listen, not from [services.state].target, so the config +// only needs the fields validation requires plus the db_path it owns. The +// forge token resolves from an env var, so no real credential is needed. +func writeMinimalConfig(t *testing.T, dir string) string { + t.Helper() + cfg := filepath.Join(dir, "config.toml") + content := fmt.Sprintf(`bot_user = "archie-bot" +db_path = %q +[forge] +type = "github" +host = "https://github.example.com" +token = { engine = "env", key = "ARCHIE_GITHUB_TOKEN" } +[[repos]] +owner = "acme" +name = "widget" +`, filepath.Join(dir, "archie.db")) + if err := os.WriteFile(cfg, []byte(content), 0o600); err != nil { + t.Fatalf("write config: %v", err) + } + return cfg +} + +// syncBuffer is a concurrency-safe buffer for a subprocess's stdout/stderr: +// the os/exec goroutine writes while the test goroutine reads (to parse the +// log and to report it on failure). A plain strings.Builder races under +// -race, so both paths take the mutex. +type syncBuffer struct { + mu sync.Mutex + b bytes.Buffer +} + +func (s *syncBuffer) Write(p []byte) (int, error) { + s.mu.Lock() + defer s.mu.Unlock() + return s.b.Write(p) +} + +func (s *syncBuffer) String() string { + s.mu.Lock() + defer s.mu.Unlock() + return s.b.String() +} + +// stateStoreProcess wraps a running archie-state-store subprocess plus the +// addresses it bound. addr is the gRPC listen address; readyAddr is the +// readiness HTTP surface. It is stopped gracefully on cleanup by SIGTERM, the +// signal the systemd unit ("Restart=on-failure") and the daemon use to drain +// the server. +type stateStoreProcess struct { + cmd *exec.Cmd + addr string + readyAddr string + log *syncBuffer +} + +// startStateStoreProcess starts bin with config, waiting for the gRPC listener +// to come up and the readiness surface to answer /healthz. It parses the +// binary's stderr log for the bound addresses because --listen 127.0.0.1:0 +// lets the process choose ephemeral ports, and the real address (not the +// configured one) is what a consumer must dial. +func startStateStoreProcess(t *testing.T, bin, cfg string) *stateStoreProcess { + t.Helper() + cmd := exec.CommandContext(t.Context(), bin, "-config", cfg, "-listen", "127.0.0.1:0", "-ready-addr", "127.0.0.1:0") + cmd.Env = append(os.Environ(), "ARCHIE_GITHUB_TOKEN=test-token") + var log syncBuffer + cmd.Stderr = &log + cmd.Stdout = &log + if err := cmd.Start(); err != nil { + t.Fatalf("start state store: %v", err) + } + p := &stateStoreProcess{cmd: cmd, log: &log} + t.Cleanup(func() { p.stop(t) }) + + // Recover the ephemeral ports from the JSON log lines the binary emits. + deadline := time.Now().Add(15 * time.Second) + for time.Now().Before(deadline) { + if p.parseAddrs() { + break + } + if cmd.ProcessState != nil { + t.Fatalf("state store exited early:\n%s", log.String()) + } + time.Sleep(25 * time.Millisecond) + } + if p.addr == "" || p.readyAddr == "" { + t.Fatalf("did not recover bound addresses from log:\n%s", log.String()) + } + + // The readiness surface is the liveness signal the runbook uses; poll it + // until it answers (the gRPC listener may come up a tick ahead of it). + deadline = time.Now().Add(15 * time.Second) + req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://"+p.readyAddr+"/healthz", nil) + if err != nil { + t.Fatalf("build readiness request: %v", err) + } + for time.Now().Before(deadline) { + if resp, err := http.DefaultClient.Do(req); err == nil { + _ = resp.Body.Close() + if resp.StatusCode == http.StatusOK { + return p + } + } + time.Sleep(25 * time.Millisecond) + } + t.Fatalf("readiness /healthz never became healthy:\n%s", log.String()) + return nil +} + +// parseAddrs scans the captured log for the bound gRPC and readiness +// addresses. The binary logs "archie-state-store running" with the gRPC addr +// and "state store readiness listening" with the HTTP addr. +func (p *stateStoreProcess) parseAddrs() bool { + if p.addr != "" && p.readyAddr != "" { + return true + } + scanner := bufio.NewScanner(strings.NewReader(p.log.String())) + for scanner.Scan() { + line := scanner.Text() + var rec map[string]any + if err := json.Unmarshal([]byte(line), &rec); err != nil { + continue + } + if msg, _ := rec["msg"].(string); msg == "archie-state-store running" { + if addr, ok := rec["addr"].(string); ok && addr != "" && p.addr == "" { + p.addr = addr + } + } + if msg, _ := rec["msg"].(string); msg == "state store readiness listening" { + if addr, ok := rec["addr"].(string); ok && addr != "" && p.readyAddr == "" { + p.readyAddr = strings.TrimPrefix(addr, "http://") + } + } + } + return p.addr != "" && p.readyAddr != "" +} + +// stop drains the process with SIGTERM and waits for it to exit. This is the +// same graceful-stop path serveStateStore's signal handling implements. +func (p *stateStoreProcess) stop(t *testing.T) { + t.Helper() + if p.cmd.Process == nil { + return + } + _ = p.cmd.Process.Signal(syscall.SIGTERM) + done := make(chan struct{}) + go func() { _ = p.cmd.Wait(); close(done) }() + select { + case <-done: + case <-time.After(10 * time.Second): + _ = p.cmd.Process.Kill() + <-done + } +} + +// contract is the combined consumer surface a daemon/gateway/agent mixes: the +// task lifecycle + events surface (store.TaskStore) and the capture surface. +// Mapping/binding/dispatch is exercised separately in +// TestStateStoreRealProcessRemoteSurfaces via the concrete Client, which +// asserts against each narrow interface at compile time. +type contract interface { + store.TaskStore + store.CaptureStore +} + +// dial opens a gRPC connection to the state store process and wraps it in the +// remote contract adapter, exactly as the daemon/gateway/agent consumer does +// (internal/infrastructure/staterpc.Client). +func dial(t *testing.T, addr string) contract { + t.Helper() + conn, err := grpc.NewClient("passthrough:///"+addr, grpc.WithTransportCredentials(insecure.NewCredentials())) + if err != nil { + t.Fatalf("dial %s: %v", addr, err) + } + t.Cleanup(func() { _ = conn.Close() }) + return staterpc.NewClient(conn) +} + +// capture builds a CapturedEvent without pulling the whole store field set +// into every call site. +func capture(source, body string) store.CapturedEvent { + return store.CapturedEvent{Source: source, Body: body, Authenticated: true} +} + +// TestStateStoreRealProcessSmoke exercises the actual archie-state-store +// binary end-to-end: the combined task/event/capture view through one consumer +// adapter, error-sentinel fidelity across the wire, the not-found-as-(nil,nil) +// convention, and single-owner SQLite (the binary owns the one archie.db file +// and the consumer dials gRPC without opening any store file). +func TestStateStoreRealProcessSmoke(t *testing.T) { + if testing.Short() { + t.Skip("real-process smoke test builds and execs the binary; skip under -short") + } + dir := t.TempDir() + bin := buildBinary(t, dir) + cfg := writeMinimalConfig(t, dir) + st := startStateStoreProcess(t, bin, cfg) + cl := dial(t, st.addr) + + ctx := t.Context() + + // Combined task/event/capture view: a task, an event on that task, and an + // unrelated capture written through one consumer, read back across all + // three surfaces in a single pass (the dashboard/observability read path). + task, err := cl.EnqueueChatTask(ctx, "acme", "widget", "verify state store", "body", "implement", "") + if err != nil || task == nil { + t.Fatalf("EnqueueChatTask = %+v, %v", task, err) + } + if err := cl.Transition(ctx, task.ID, task.Status, "running", "smoke started"); err != nil { + t.Fatalf("Transition: %v", err) + } + if _, err := cl.InsertEvent(ctx, events.Event{Kind: "stage_finish", TaskID: task.ID, Data: map[string]any{"duration_ms": 12.0}}); err != nil { + t.Fatalf("InsertEvent: %v", err) + } + if capID, err := cl.InsertCapture(ctx, capture("sentry", `{"id":1}`), 0, 0); err != nil || capID == 0 { + t.Fatalf("InsertCapture = (%d, %v)", capID, err) + } + + got, err := cl.TaskByID(ctx, task.ID) + if err != nil || got == nil || got.ID != task.ID || got.Status != "running" { + t.Fatalf("TaskByID after transition = %+v, %v", got, err) + } + if evs, err := cl.TaskEvents(ctx, task.ID); err != nil || len(evs) != 1 || evs[0].Kind != "stage_finish" { + t.Fatalf("TaskEvents = %+v, %v", evs, err) + } + if caps, err := cl.ListCaptures(ctx, 10); err != nil || len(caps) != 1 || caps[0].Source != "sentry" { + t.Fatalf("ListCaptures = %+v, %v", caps, err) + } + if counts, err := cl.StatusCounts(ctx); err != nil || counts["running"] != 1 { + t.Fatalf("StatusCounts = %+v, %v", counts, err) + } + + t.Run("error_sentinel_fidelity", func(t *testing.T) { + // A stale transition must rehydrate to store.ErrStaleTransition across + // the wire (§7). Because the binary is up and we dialed it, a sentinel + // mismatch here would surface as a transport error, so a pass proves + // the gRPC mapError/unmapError round-trip, not just the in-process + // adapter. + err := cl.Transition(ctx, task.ID, "queued", "merged", "") + if !errors.Is(err, store.ErrStaleTransition) { + t.Fatalf("stale Transition = %v, want store.ErrStaleTransition", err) + } + }) + + t.Run("not_found_is_not_error", func(t *testing.T) { + // Reading a missing task returns (nil, nil), never a NotFound gRPC + // error (§7's found=false-not-error convention). + missing, err := cl.TaskByID(ctx, task.ID+999999) + if err != nil || missing != nil { + t.Fatalf("TaskByID missing = (%+v, %v), want (nil, nil)", missing, err) + } + }) + + t.Run("single_owner_sqlite", func(t *testing.T) { + // The binary owns one task SQLite file (db_path + "-tasks.sqlite"); + // the consumer called only gRPC methods and never opened a store file. + // The config anchor path itself must not be opened directly (no dual + // store ownership, §12 step 8). + taskFile := filepath.Join(dir, "archie.db-tasks.sqlite") + if _, err := os.Stat(taskFile); err != nil { + t.Fatalf("state store did not own %s: %v", taskFile, err) + } + if _, err := os.Stat(filepath.Join(dir, "archie.db")); err == nil { + t.Fatalf("config anchor path was opened directly; dual store ownership violated §12 step 8") + } + }) +} + +// TestStateStoreRealProcessDeadline verifies the per-call deadline behaviour +// the state store contract requires (§6 StoreTimeout): a consumer context that +// is cancelled before the call completes must surface a deadline/cancel error +// rather than hang or silently succeed. +func TestStateStoreRealProcessDeadline(t *testing.T) { + if testing.Short() { + t.Skip("real-process smoke test builds and execs the binary; skip under -short") + } + dir := t.TempDir() + bin := buildBinary(t, dir) + cfg := writeMinimalConfig(t, dir) + st := startStateStoreProcess(t, bin, cfg) + cl := dial(t, st.addr) + + ctx, cancel := context.WithCancel(t.Context()) + cancel() + _, err := cl.StatusCounts(ctx) + if err == nil { + t.Fatal("StatusCounts on a cancelled context returned nil; expected a deadline/cancel error") + } + if !errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("StatusCounts cancelled = %v, want context.Canceled or DeadlineExceeded", err) + } +} + +// TestStateStoreRealProcessRestartRecovery starts the process, writes a task, +// stops it (SIGTERM), restarts it on the same config/db_path, and verifies the +// task and its events were recovered from the on-disk SQLite -- the +// restart/recovery path the runbook relies on when archie-state-store is +// restarted as a service. +func TestStateStoreRealProcessRestartRecovery(t *testing.T) { + if testing.Short() { + t.Skip("real-process smoke test builds and execs the binary; skip under -short") + } + dir := t.TempDir() + bin := buildBinary(t, dir) + cfg := writeMinimalConfig(t, dir) + + first := startStateStoreProcess(t, bin, cfg) + s1 := dial(t, first.addr) + task, err := s1.EnqueueChatTask(t.Context(), "acme", "widget", "survive restart", "body", "implement", "") + if err != nil || task == nil { + t.Fatalf("first EnqueueChatTask = %+v, %v", task, err) + } + if _, err := s1.InsertEvent(t.Context(), events.Event{Kind: "run_started", TaskID: task.ID}); err != nil { + t.Fatalf("first InsertEvent: %v", err) + } + first.stop(t) + + // Restart on the same config; the SQLite file persists on disk. + second := startStateStoreProcess(t, bin, cfg) + s2 := dial(t, second.addr) + + got, err := s2.TaskByID(t.Context(), task.ID) + if err != nil || got == nil || got.Title != "survive restart" { + t.Fatalf("TaskByID after restart = %+v, %v", got, err) + } + if evs, err := s2.TaskEvents(t.Context(), task.ID); err != nil || len(evs) != 1 { + t.Fatalf("TaskEvents after restart = %+v, %v", evs, err) + } + // Stale-running work reopened on a fresh process is recovered by + // RecoverStale, which must return without error on the restarted store. + if _, err := s2.RecoverStale(t.Context()); err != nil { + t.Fatalf("RecoverStale after restart: %v", err) + } +} + +// TestStateStoreRealProcessRemoteSurfaces exercises the capture/mapping/ +// binding surfaces the standalone process fronts (the daemon/webui surfaces +// swapped to the adapter in .4.4/.4.5), proving one process serves the full +// combined contract a consumer mixes. +func TestStateStoreRealProcessRemoteSurfaces(t *testing.T) { + if testing.Short() { + t.Skip("real-process smoke test builds and execs the binary; skip under -short") + } + dir := t.TempDir() + bin := buildBinary(t, dir) + cfg := writeMinimalConfig(t, dir) + st := startStateStoreProcess(t, bin, cfg) + + conn, err := grpc.NewClient("passthrough:///"+st.addr, grpc.WithTransportCredentials(insecure.NewCredentials())) + if err != nil { + t.Fatalf("dial: %v", err) + } + t.Cleanup(func() { _ = conn.Close() }) + cl := staterpc.NewClient(conn) + + ctx := t.Context() + mappingID, err := cl.InsertMapping(ctx, mapping.Mapping{Name: "m1", Fields: []mapping.Field{{Name: "a", Path: "a", Type: mapping.TypeString}}}) + if err != nil || mappingID == 0 { + t.Fatalf("InsertMapping = (%d, %v)", mappingID, err) + } + if got, err := cl.GetMapping(ctx, mappingID); err != nil || got == nil || got.Name != "m1" { + t.Fatalf("GetMapping = %+v, %v", got, err) + } + bindingID, err := cl.InsertBinding(ctx, binding.Binding{ + Name: "b1", Matcher: binding.Matcher{Source: "sentry"}, MappingID: mappingID, + Workflow: "implement", Secret: "0123456789abcdef0123456789abcdef", + }) + if err != nil || bindingID == 0 { + t.Fatalf("InsertBinding = (%d, %v)", bindingID, err) + } + // A binding starts as draft and must be edited (draft -> pending_approval) + // then approved (pending_approval -> armed) before it is armed for a + // source. Drive the state machine so ArmedBindingsForSource returns it. + if err := cl.UpdateBinding(ctx, binding.Binding{ + ID: bindingID, Name: "b1", Matcher: binding.Matcher{Source: "sentry"}, + MappingID: mappingID, Workflow: "implement", Owner: "acme", Repo: "widget", + Secret: "0123456789abcdef0123456789abcdef", + }); err != nil { + t.Fatalf("UpdateBinding: %v", err) + } + if err := cl.ApproveBinding(ctx, bindingID); err != nil { + t.Fatalf("ApproveBinding: %v", err) + } + if armed, err := cl.ArmedBindingsForSource(ctx, "sentry"); err != nil || len(armed) != 1 || armed[0].ID != bindingID { + t.Fatalf("ArmedBindingsForSource = %+v, %v", armed, err) + } + // Remote surfaces share error-sentinel fidelity on the wire too. + missing := mapping.Mapping{ID: mappingID + 999999, Name: "x", Fields: []mapping.Field{{Name: "a", Path: "a", Type: mapping.TypeString}}} + if err := cl.UpdateMapping(ctx, missing); !errors.Is(err, store.ErrMappingNotFound) { + t.Fatalf("UpdateMapping missing = %v, want ErrMappingNotFound", err) + } +} diff --git a/deployments/README.md b/deployments/README.md index a1eace034..351f0ec4b 100644 --- a/deployments/README.md +++ b/deployments/README.md @@ -61,3 +61,22 @@ archie-state-store -config ~/.config/archie/config.toml -listen 127.0.0.1:9090 # or container-mode: bind the bridge gateway with a token archie-state-store -config ~/.config/archie/config.toml -listen 172.17.0.1:9090 -token ``` + +## Verifying the State Store + +`archie-state-store` exposes an optional HTTP readiness surface when launched +with `-ready-addr` (the systemd unit uses `127.0.0.1:9091`). Use it to confirm +the process is up and the store is healthy before starting `archied`: + +```bash +curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9091/healthz # 200 = process up +curl -s http://127.0.0.1:9091/health/detailed # state_db component -> 200, or 503 when degraded +``` + +A running State Store owns exactly one task SQLite file (`-tasks.sqlite`); +`archied` and `archie-gateway` never open `archie.db` directly, so a second +store process on the same `db_path`, or a consumer that opens the task file, +indicates dual-store ownership (`docs/prds/state-store-contract.md` §12 step 7/8). +The readiness probe is the single-writer check the runbook has against a +degraded store: a `503` from `/health/detailed` means `archied` should be +started only after the store is healthy. diff --git a/internal/app/archied/state_store_client_test.go b/internal/app/archied/state_store_client_test.go index 50efa53a6..9c6442ee6 100644 --- a/internal/app/archied/state_store_client_test.go +++ b/internal/app/archied/state_store_client_test.go @@ -80,3 +80,29 @@ func TestOpenStateStoreAdapterRemoteFailsClosed(t *testing.T) { t.Fatal("non-loopback without a token should fail closed") } } + +// TestStateStoreSingleOwnerComposition proves the single-owner SQLite +// invariant at the composition root: both production consumers (the daemon +// Run path and the gateway RunGateway) resolve their task-store surface from +// openStateStoreAdapter, which REQUIRES [services.state].target and never +// opens archie.db directly. After .4.6 the in-process store path is deleted +// (docs/prds/state-store-contract.md §12 step 7), so an empty target is a +// startup error rather than the old local default -- if it were not, the +// daemon or gateway would open the single SQLite file behind the store +// service's back, violating no-dual-store-ownership. +func TestStateStoreSingleOwnerComposition(t *testing.T) { + b := newBootstrap() + b.secrets = &secret.Registry{} + + // The daemon openStores path must not construct a local *store.Store. + // openStores only resolves secrets and the forge client, then opens chat + // sessions; the task store surface comes from the remote adapter, so no + // archie.db task file is touched. This asserts openStateStoreAdapter (the + // only task-store constructor) rejects an empty target. + if err := b.openStateStoreAdapter(); err == nil { + t.Fatal("empty services.state.target must error: the in-process store path is deleted (single owner = archie-state-store)") + } + if b.stateStore != nil { + t.Fatalf("stateStore = %p, want nil (no local store opened)", b.stateStore) + } +} diff --git a/internal/infrastructure/staterpc/error_test.go b/internal/infrastructure/staterpc/error_test.go new file mode 100644 index 000000000..913620274 --- /dev/null +++ b/internal/infrastructure/staterpc/error_test.go @@ -0,0 +1,127 @@ +package staterpc + +import ( + "context" + "errors" + "strings" + "testing" + + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" + + "github.com/samcharles93/archie-core/internal/store" +) + +// TestUnmapErrorPreservesContextIdentity pins the §6 deadline contract that +// unmapError must rehydrate a gRPC Canceled / DeadlineExceeded status back to +// the standard context sentinels. The agent's deadlineStore bounds each Store +// call with context.WithTimeout and the workflow consumer checks +// errors.Is(err, context.DeadlineExceeded) to mark a stage interrupted rather +// than failed (workflow.go); if unmapError folded those into a sanitised +// internal error, a merely interrupted stage would be parked as a failure. +func TestUnmapErrorPreservesContextIdentity(t *testing.T) { + tests := []struct { + name string + err error + want error + }{ + {name: "cancelled", err: status.Error(codes.Canceled, context.Canceled.Error()), want: context.Canceled}, + {name: "deadline exceeded", err: status.Error(codes.DeadlineExceeded, context.DeadlineExceeded.Error()), want: context.DeadlineExceeded}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := unmapError(tt.err) + if !errors.Is(got, tt.want) { + t.Fatalf("unmapError(%q) = %v, want errors.Is(err, %v)", tt.err, got, tt.want) + } + if errors.Is(got, store.ErrStaleTransition) { + t.Fatalf("unmapError(%q) = %v, must not map to a store sentinel", tt.err, got) + } + }) + } +} + +// TestMapErrorPreservesContextIdentity pins the server side of the same §6 +// contract: if the store (or a server handler) returns a context error because +// the caller's ctx expired mid-call, mapError must map it to the matching +// gRPC code rather than sanitising it into codes.Internal. Otherwise the +// client never sees the deadline identity and an interrupted stage is wrongly +// treated as a failure. +func TestMapErrorPreservesContextIdentity(t *testing.T) { + tests := []struct { + name string + err error + want error + }{ + {name: "cancelled", err: context.Canceled, want: context.Canceled}, + {name: "deadline exceeded", err: context.DeadlineExceeded, want: context.DeadlineExceeded}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + mapped := mapError(tt.err) + if got := unmapError(mapped); !errors.Is(got, tt.want) { + t.Fatalf("mapError/unmapError round-trip for %q = %v, want errors.Is(err, %v)", tt.err, got, tt.want) + } + }) + } +} + +// TestUnmapErrorSentinelFidelity pins the §7 sentinel rehydration round-trip +// for each store error code, so a consumer's errors.Is(err, store.ErrX) keeps +// working across the wire. +func TestUnmapErrorSentinelFidelity(t *testing.T) { + tests := []struct { + name string + store error + rehyd error + }{ + {name: "stale transition", store: store.ErrStaleTransition, rehyd: store.ErrStaleTransition}, + {name: "binding not found", store: store.ErrBindingNotFound, rehyd: store.ErrBindingNotFound}, + {name: "mapping not found", store: store.ErrMappingNotFound, rehyd: store.ErrMappingNotFound}, + {name: "binding overlap", store: store.ErrBindingOverlap, rehyd: store.ErrBindingOverlap}, + {name: "binding transition", store: store.ErrBindingTransition, rehyd: store.ErrBindingTransition}, + {name: "already dispatched", store: store.ErrAlreadyDispatched, rehyd: store.ErrAlreadyDispatched}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + // mapError converts the store sentinel to a gRPC status; unmapError + // must recover the same sentinel. This is the round-trip the wire + // contract guarantees (§7). + mapped := mapError(tt.store) + got := unmapError(mapped) + if !errors.Is(got, tt.rehyd) { + t.Fatalf("round-trip %q -> %q: got %v, want %v", tt.store, mapped, got, tt.rehyd) + } + }) + } +} + +// TestUnmapErrorDoesNotLeakInternalDetail pins the §7 sanitisation rule: an +// infra/unknown error maps to codes.Internal and must not rehydrate to a store +// sentinel, the public message must not carry the raw wrapped detail (SQL +// paths, provider details), and a non-status error is returned unchanged. +func TestUnmapErrorDoesNotLeakInternalDetail(t *testing.T) { + const detail = "sql: path=/var/lib/archie/archie.db secret=abcdef" + internal := mapError(errors.New(detail)) + got := unmapError(internal) + if errors.Is(got, store.ErrStaleTransition) || errors.Is(got, store.ErrBindingNotFound) || errors.Is(got, store.ErrAlreadyDispatched) { + t.Fatalf("unmapError(internal) = %v, must not map to a store sentinel", got) + } + if st, ok := status.FromError(got); !ok || st.Code() != codes.Internal { + t.Fatalf("unmapError(internal) = %v, want a codes.Internal status", got) + } + if strings.Contains(got.Error(), detail) { + t.Fatalf("unmapError leaked raw internal detail: %q", got.Error()) + } + + // A raw (non-status) error is returned unchanged (not wrapped into a + // status or a store sentinel). + raw := errors.New("transport failure") + got = unmapError(raw) + if errors.Is(got, store.ErrStaleTransition) { + t.Fatalf("unmapError(raw) = %v, must not map to a store sentinel", got) + } + if !errors.Is(got, raw) { + t.Fatalf("unmapError(raw) = %v, want an error chain including the original", got) + } +} diff --git a/internal/infrastructure/staterpc/values.go b/internal/infrastructure/staterpc/values.go index 9128c4cac..ea4793e43 100644 --- a/internal/infrastructure/staterpc/values.go +++ b/internal/infrastructure/staterpc/values.go @@ -1,6 +1,7 @@ package staterpc import ( + "context" "encoding/json" "errors" "fmt" @@ -251,10 +252,24 @@ const ( // (which may contain SQL, provider detail, or secrets). Infra errors map to // codes.Internal with a sanitised message; the caller is expected to log the // full error server-side before calling mapError. +// +// A context cancellation/deadline raised by the store (or by a caller ctx that +// expires mid-call, §6) must retain its identity, not be folded into +// codes.Internal: gRPC-Go surfaces context.Canceled/DeadlineExceeded to the +// client only for those exact codes, and the agent's workflow consumer +// depends on errors.Is(err, context.DeadlineExceeded) to distinguish an +// interrupted stage from a failed one. So a context error maps to its own +// gRPC code (and unmapError rehydrates it back to the sentinel). func mapError(err error) error { if err == nil { return nil } + if errors.Is(err, context.Canceled) { + return status.Error(codes.Canceled, context.Canceled.Error()) + } + if errors.Is(err, context.DeadlineExceeded) { + return status.Error(codes.DeadlineExceeded, context.DeadlineExceeded.Error()) + } switch { case errors.Is(err, store.ErrStaleTransition): return status.Error(codes.FailedPrecondition, msgStaleTransition) @@ -275,8 +290,17 @@ func mapError(err error) error { // unmapError rehydrates a gRPC status error back to the store sentinel it // came from, so a caller's errors.Is(err, store.ErrX) keeps working across -// the wire. A non-status error (e.g. a transport failure, a cancelled or -// deadline-exceeded context) is returned unchanged. +// the wire. A non-status error (e.g. a transport failure) is returned +// unchanged. +// +// The deadline/cancel identity must survive too (§6): the agent's +// deadlineStore bounds each Store call with context.WithTimeout, and the +// workflow consumer checks errors.Is(err, context.DeadlineExceeded) to +// decide whether a stage was interrupted by shutdown rather than failed +// (workflow.go). gRPC-Go surfaces an expired or cancelled context as a +// *status.Error whose code is DeadlineExceeded or Canceled, so we rehydrate +// those back to the standard context sentinels -- otherwise the consumer +// would (wrongly) park a task that was merely interrupted. func unmapError(err error) error { if err == nil { return nil @@ -286,6 +310,10 @@ func unmapError(err error) error { return err } switch st.Code() { + case codes.Canceled: + return context.Canceled + case codes.DeadlineExceeded: + return context.DeadlineExceeded case codes.FailedPrecondition: switch st.Message() { case msgStaleTransition: