test(staterpc): verify State Store end-to-end and preserve context error identity (#775)
* test(state-store): real-process smoke + deadline/sentinel error contract Add end-to-end verification for archie-core-8cda.4.7, the .4.7 State Store verification stage against docs/prds/state-store-contract.md (rev. 2c): - Real-process smoke coverage: build ./cmd/archie-state-store, run it against a minimal config on loopback, poll the readiness surface, and drive the State Store gRPC contract through staterpc.NewClient exactly as a consumer does. Covers the combined task/event/capture view, error-sentinel fidelity across the wire, the found=false-not-error convention, single-owner SQLite (the binary owns the one archie.db file; the consumer never opens a store file), per-call deadline behaviour, restart/recovery (SIGTERM + reopen on the same db), and the remote capture/mapping/binding surfaces. - Fix a real deadline-identity gap in unmapError: a cancelled or deadline-exceeded context surfaced by gRPC-Go as a *status.Error was re-wrapped into a sanitised internal-style error, breaking errors.Is(err, context.Canceled/DeadlineExceeded). The agent's deadlineStore bounds each Store call with context.WithTimeout and the workflow consumer (workflow.go) uses errors.Is(err, context.DeadlineExceeded) to mark a stage interrupted rather than failed, so the re-wrap would wrongly park an interrupted task. Unmap Canceled/DeadlineExceeded codes back to the standard context sentinels. - Pin the error-mapping contract in staterpc: context-identity preservation, sentinel round-trip fidelity for every store sentinel, and the sanitisation rule (an Internal error must not leak raw detail or map to a sentinel). * test(state-store): assert single-owner SQLite composition (daemon/gateway require target) The single-owner invariant (docs/prds/state-store-contract.md §12 step 7/8) is that only archie-state-store opens archie.db; the daemon and gateway resolve their task-store surface from openStateStoreAdapter, which requires [services.state].target and opens no local store. This test pins that composition root so a future change cannot silently reintroduce a local *store.Store in the daemon or gateway path (dual-store ownership). * test(state-store): skip real-process smoke tests under -short The smoke tests build and exec the real archie-state-store binary, which is slow enough that a -short run (e.g. a focused edit loop) should skip them. task check (go test ./... -count=1) does not use -short, so they still run in the full gate. * test(state-store): fix subprocess log race under -race The smoke test captured the subprocess's stdout/stderr in a strings.Builder that the os/exec writer goroutine and the test reader goroutine touched concurrently, racing under -race. Replace it with a mutex-guarded buffer, the standard pattern for reading a running subprocess's output. * test(state-store): satisfy lint (noctx, errorlint) on new tests - exec.Command -> exec.CommandContext, http.Get -> NewRequestWithContext + Client.Do, so the subprocess and readiness polling honour the test context (noctx). - errorlint: compare the raw non-status error via errors.Is rather than != on a wrapped error. * docs(deployments): document State Store readiness verification and single-owner invariant The deployment runbook described how to launch archie-state-store but not how to verify it is healthy before starting archied. Document the -ready-addr readiness surface (/healthz, /health/detailed state_db component) and the single-owner invariant: one task SQLite file owned by archie-state-store, with archied/archie-gateway never opening archie.db directly (state-store-contract §12 step 7/8). * fix(state-store): preserve context deadline/cancel identity in mapError The symmetric server-side gap to the unmapError fix: if a store query fails because the caller's context expires mid-call (§6 StoreTimeout), the handler's mapError folded context.Canceled/DeadlineExceeded into codes.Internal. The client would receive a sanitised internal error with no deadline identity, so the agent's workflow consumer (errors.Is(err, context.DeadlineExceeded)) could not distinguish an interrupted stage from a failed one. Map a context cancel/deadline error to its matching gRPC code so unmapError rehydrates it back to the standard context sentinel, completing the §6/§7 round-trip. Pin both directions with TestMapErrorPreservesContextIdentity.
This commit is contained in:
1 parent
3a0dfa924d
commit
33751abe4e
5 files changed
+680
-2
No files matched your search
@@ -0,0 +1,478 @@
|
||||
// Package main smoke-tests the standalone archie-state-store process by
|
||||
// building the real binary, running it against a minimal config on the
|
||||
// loopback interface, and driving it through the State Store gRPC contract
|
||||
// exactly the way a consumer (archied, archie-gateway, archie-agent) does.
|
||||
//
|
||||
// This is the .4.7 end-to-end verification for
|
||||
// docs/prds/state-store-contract.md (rev. 2c): it proves the process owns the
|
||||
// single archie.db task SQLite, serves the combined task/event/capture
|
||||
// surface over one gRPC service, preserves error-sentinel fidelity across the
|
||||
// wire, honours a per-call deadline, survives a restart/recovery cycle, and
|
||||
// is the sole owner of the SQLite file. See §5 (binary layout), §6 (handoff),
|
||||
// §7 (error semantics), §11 (remote adapter) and §12 step 7/8 (single owner).
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"google.golang.org/grpc"
|
||||
"google.golang.org/grpc/credentials/insecure"
|
||||
|
||||
"github.com/samcharles93/archie-core/internal/domain/binding"
|
||||
"github.com/samcharles93/archie-core/internal/domain/mapping"
|
||||
"github.com/samcharles93/archie-core/internal/events"
|
||||
"github.com/samcharles93/archie-core/internal/infrastructure/staterpc"
|
||||
"github.com/samcharles93/archie-core/internal/store"
|
||||
)
|
||||
|
||||
// repoRoot walks up from the test's source directory to find the module root
|
||||
// (where go.mod lives). The test runs in the package directory
|
||||
// (cmd/archie-state-store), so the module root is above it; walking keeps the
|
||||
// result correct regardless of the harness working directory.
|
||||
func repoRoot(t *testing.T) string {
|
||||
t.Helper()
|
||||
_, file, _, ok := runtime.Caller(0)
|
||||
if !ok {
|
||||
t.Fatal("cannot determine test source path")
|
||||
}
|
||||
dir := filepath.Dir(file)
|
||||
for {
|
||||
if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil {
|
||||
return dir
|
||||
}
|
||||
parent := filepath.Dir(dir)
|
||||
if parent == dir {
|
||||
t.Fatal("go.mod not found above test source")
|
||||
}
|
||||
dir = parent
|
||||
}
|
||||
}
|
||||
|
||||
// buildBinary compiles ./cmd/archie-state-store into dir and returns the path.
|
||||
// This is the real binary the smoke test exercises, not a test double -- the
|
||||
// point of .4.7 is to verify the actual process.
|
||||
func buildBinary(t *testing.T, dir string) string {
|
||||
t.Helper()
|
||||
bin := filepath.Join(dir, "archie-state-store")
|
||||
cmd := exec.CommandContext(t.Context(), "go", "build", "-o", bin, "./cmd/archie-state-store")
|
||||
cmd.Dir = repoRoot(t)
|
||||
cmd.Env = os.Environ()
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("go build archie-state-store: %v\n%s", err, out)
|
||||
}
|
||||
if _, err := os.Stat(bin); err != nil {
|
||||
t.Fatalf("built binary not found: %v", err)
|
||||
}
|
||||
return bin
|
||||
}
|
||||
|
||||
// writeMinimalConfig writes a config.toml that passes configuration.Validate
|
||||
// while keeping the process minimal: the standalone binary reads its gRPC
|
||||
// listen address from -listen, not from [services.state].target, so the config
|
||||
// only needs the fields validation requires plus the db_path it owns. The
|
||||
// forge token resolves from an env var, so no real credential is needed.
|
||||
func writeMinimalConfig(t *testing.T, dir string) string {
|
||||
t.Helper()
|
||||
cfg := filepath.Join(dir, "config.toml")
|
||||
content := fmt.Sprintf(`bot_user = "archie-bot"
|
||||
db_path = %q
|
||||
[forge]
|
||||
type = "github"
|
||||
host = "https://github.example.com"
|
||||
token = { engine = "env", key = "ARCHIE_GITHUB_TOKEN" }
|
||||
[[repos]]
|
||||
owner = "acme"
|
||||
name = "widget"
|
||||
`, filepath.Join(dir, "archie.db"))
|
||||
if err := os.WriteFile(cfg, []byte(content), 0o600); err != nil {
|
||||
t.Fatalf("write config: %v", err)
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
|
||||
// syncBuffer is a concurrency-safe buffer for a subprocess's stdout/stderr:
|
||||
// the os/exec goroutine writes while the test goroutine reads (to parse the
|
||||
// log and to report it on failure). A plain strings.Builder races under
|
||||
// -race, so both paths take the mutex.
|
||||
type syncBuffer struct {
|
||||
mu sync.Mutex
|
||||
b bytes.Buffer
|
||||
}
|
||||
|
||||
func (s *syncBuffer) Write(p []byte) (int, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.b.Write(p)
|
||||
}
|
||||
|
||||
func (s *syncBuffer) String() string {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.b.String()
|
||||
}
|
||||
|
||||
// stateStoreProcess wraps a running archie-state-store subprocess plus the
|
||||
// addresses it bound. addr is the gRPC listen address; readyAddr is the
|
||||
// readiness HTTP surface. It is stopped gracefully on cleanup by SIGTERM, the
|
||||
// signal the systemd unit ("Restart=on-failure") and the daemon use to drain
|
||||
// the server.
|
||||
type stateStoreProcess struct {
|
||||
cmd *exec.Cmd
|
||||
addr string
|
||||
readyAddr string
|
||||
log *syncBuffer
|
||||
}
|
||||
|
||||
// startStateStoreProcess starts bin with config, waiting for the gRPC listener
|
||||
// to come up and the readiness surface to answer /healthz. It parses the
|
||||
// binary's stderr log for the bound addresses because --listen 127.0.0.1:0
|
||||
// lets the process choose ephemeral ports, and the real address (not the
|
||||
// configured one) is what a consumer must dial.
|
||||
func startStateStoreProcess(t *testing.T, bin, cfg string) *stateStoreProcess {
|
||||
t.Helper()
|
||||
cmd := exec.CommandContext(t.Context(), bin, "-config", cfg, "-listen", "127.0.0.1:0", "-ready-addr", "127.0.0.1:0")
|
||||
cmd.Env = append(os.Environ(), "ARCHIE_GITHUB_TOKEN=test-token")
|
||||
var log syncBuffer
|
||||
cmd.Stderr = &log
|
||||
cmd.Stdout = &log
|
||||
if err := cmd.Start(); err != nil {
|
||||
t.Fatalf("start state store: %v", err)
|
||||
}
|
||||
p := &stateStoreProcess{cmd: cmd, log: &log}
|
||||
t.Cleanup(func() { p.stop(t) })
|
||||
|
||||
// Recover the ephemeral ports from the JSON log lines the binary emits.
|
||||
deadline := time.Now().Add(15 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
if p.parseAddrs() {
|
||||
break
|
||||
}
|
||||
if cmd.ProcessState != nil {
|
||||
t.Fatalf("state store exited early:\n%s", log.String())
|
||||
}
|
||||
time.Sleep(25 * time.Millisecond)
|
||||
}
|
||||
if p.addr == "" || p.readyAddr == "" {
|
||||
t.Fatalf("did not recover bound addresses from log:\n%s", log.String())
|
||||
}
|
||||
|
||||
// The readiness surface is the liveness signal the runbook uses; poll it
|
||||
// until it answers (the gRPC listener may come up a tick ahead of it).
|
||||
deadline = time.Now().Add(15 * time.Second)
|
||||
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://"+p.readyAddr+"/healthz", nil)
|
||||
if err != nil {
|
||||
t.Fatalf("build readiness request: %v", err)
|
||||
}
|
||||
for time.Now().Before(deadline) {
|
||||
if resp, err := http.DefaultClient.Do(req); err == nil {
|
||||
_ = resp.Body.Close()
|
||||
if resp.StatusCode == http.StatusOK {
|
||||
return p
|
||||
}
|
||||
}
|
||||
time.Sleep(25 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("readiness /healthz never became healthy:\n%s", log.String())
|
||||
return nil
|
||||
}
|
||||
|
||||
// parseAddrs scans the captured log for the bound gRPC and readiness
|
||||
// addresses. The binary logs "archie-state-store running" with the gRPC addr
|
||||
// and "state store readiness listening" with the HTTP addr.
|
||||
func (p *stateStoreProcess) parseAddrs() bool {
|
||||
if p.addr != "" && p.readyAddr != "" {
|
||||
return true
|
||||
}
|
||||
scanner := bufio.NewScanner(strings.NewReader(p.log.String()))
|
||||
for scanner.Scan() {
|
||||
line := scanner.Text()
|
||||
var rec map[string]any
|
||||
if err := json.Unmarshal([]byte(line), &rec); err != nil {
|
||||
continue
|
||||
}
|
||||
if msg, _ := rec["msg"].(string); msg == "archie-state-store running" {
|
||||
if addr, ok := rec["addr"].(string); ok && addr != "" && p.addr == "" {
|
||||
p.addr = addr
|
||||
}
|
||||
}
|
||||
if msg, _ := rec["msg"].(string); msg == "state store readiness listening" {
|
||||
if addr, ok := rec["addr"].(string); ok && addr != "" && p.readyAddr == "" {
|
||||
p.readyAddr = strings.TrimPrefix(addr, "http://")
|
||||
}
|
||||
}
|
||||
}
|
||||
return p.addr != "" && p.readyAddr != ""
|
||||
}
|
||||
|
||||
// stop drains the process with SIGTERM and waits for it to exit. This is the
|
||||
// same graceful-stop path serveStateStore's signal handling implements.
|
||||
func (p *stateStoreProcess) stop(t *testing.T) {
|
||||
t.Helper()
|
||||
if p.cmd.Process == nil {
|
||||
return
|
||||
}
|
||||
_ = p.cmd.Process.Signal(syscall.SIGTERM)
|
||||
done := make(chan struct{})
|
||||
go func() { _ = p.cmd.Wait(); close(done) }()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(10 * time.Second):
|
||||
_ = p.cmd.Process.Kill()
|
||||
<-done
|
||||
}
|
||||
}
|
||||
|
||||
// contract is the combined consumer surface a daemon/gateway/agent mixes: the
|
||||
// task lifecycle + events surface (store.TaskStore) and the capture surface.
|
||||
// Mapping/binding/dispatch is exercised separately in
|
||||
// TestStateStoreRealProcessRemoteSurfaces via the concrete Client, which
|
||||
// asserts against each narrow interface at compile time.
|
||||
type contract interface {
|
||||
store.TaskStore
|
||||
store.CaptureStore
|
||||
}
|
||||
|
||||
// dial opens a gRPC connection to the state store process and wraps it in the
|
||||
// remote contract adapter, exactly as the daemon/gateway/agent consumer does
|
||||
// (internal/infrastructure/staterpc.Client).
|
||||
func dial(t *testing.T, addr string) contract {
|
||||
t.Helper()
|
||||
conn, err := grpc.NewClient("passthrough:///"+addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
|
||||
if err != nil {
|
||||
t.Fatalf("dial %s: %v", addr, err)
|
||||
}
|
||||
t.Cleanup(func() { _ = conn.Close() })
|
||||
return staterpc.NewClient(conn)
|
||||
}
|
||||
|
||||
// capture builds a CapturedEvent without pulling the whole store field set
|
||||
// into every call site.
|
||||
func capture(source, body string) store.CapturedEvent {
|
||||
return store.CapturedEvent{Source: source, Body: body, Authenticated: true}
|
||||
}
|
||||
|
||||
// TestStateStoreRealProcessSmoke exercises the actual archie-state-store
|
||||
// binary end-to-end: the combined task/event/capture view through one consumer
|
||||
// adapter, error-sentinel fidelity across the wire, the not-found-as-(nil,nil)
|
||||
// convention, and single-owner SQLite (the binary owns the one archie.db file
|
||||
// and the consumer dials gRPC without opening any store file).
|
||||
func TestStateStoreRealProcessSmoke(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("real-process smoke test builds and execs the binary; skip under -short")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
bin := buildBinary(t, dir)
|
||||
cfg := writeMinimalConfig(t, dir)
|
||||
st := startStateStoreProcess(t, bin, cfg)
|
||||
cl := dial(t, st.addr)
|
||||
|
||||
ctx := t.Context()
|
||||
|
||||
// Combined task/event/capture view: a task, an event on that task, and an
|
||||
// unrelated capture written through one consumer, read back across all
|
||||
// three surfaces in a single pass (the dashboard/observability read path).
|
||||
task, err := cl.EnqueueChatTask(ctx, "acme", "widget", "verify state store", "body", "implement", "")
|
||||
if err != nil || task == nil {
|
||||
t.Fatalf("EnqueueChatTask = %+v, %v", task, err)
|
||||
}
|
||||
if err := cl.Transition(ctx, task.ID, task.Status, "running", "smoke started"); err != nil {
|
||||
t.Fatalf("Transition: %v", err)
|
||||
}
|
||||
if _, err := cl.InsertEvent(ctx, events.Event{Kind: "stage_finish", TaskID: task.ID, Data: map[string]any{"duration_ms": 12.0}}); err != nil {
|
||||
t.Fatalf("InsertEvent: %v", err)
|
||||
}
|
||||
if capID, err := cl.InsertCapture(ctx, capture("sentry", `{"id":1}`), 0, 0); err != nil || capID == 0 {
|
||||
t.Fatalf("InsertCapture = (%d, %v)", capID, err)
|
||||
}
|
||||
|
||||
got, err := cl.TaskByID(ctx, task.ID)
|
||||
if err != nil || got == nil || got.ID != task.ID || got.Status != "running" {
|
||||
t.Fatalf("TaskByID after transition = %+v, %v", got, err)
|
||||
}
|
||||
if evs, err := cl.TaskEvents(ctx, task.ID); err != nil || len(evs) != 1 || evs[0].Kind != "stage_finish" {
|
||||
t.Fatalf("TaskEvents = %+v, %v", evs, err)
|
||||
}
|
||||
if caps, err := cl.ListCaptures(ctx, 10); err != nil || len(caps) != 1 || caps[0].Source != "sentry" {
|
||||
t.Fatalf("ListCaptures = %+v, %v", caps, err)
|
||||
}
|
||||
if counts, err := cl.StatusCounts(ctx); err != nil || counts["running"] != 1 {
|
||||
t.Fatalf("StatusCounts = %+v, %v", counts, err)
|
||||
}
|
||||
|
||||
t.Run("error_sentinel_fidelity", func(t *testing.T) {
|
||||
// A stale transition must rehydrate to store.ErrStaleTransition across
|
||||
// the wire (§7). Because the binary is up and we dialed it, a sentinel
|
||||
// mismatch here would surface as a transport error, so a pass proves
|
||||
// the gRPC mapError/unmapError round-trip, not just the in-process
|
||||
// adapter.
|
||||
err := cl.Transition(ctx, task.ID, "queued", "merged", "")
|
||||
if !errors.Is(err, store.ErrStaleTransition) {
|
||||
t.Fatalf("stale Transition = %v, want store.ErrStaleTransition", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("not_found_is_not_error", func(t *testing.T) {
|
||||
// Reading a missing task returns (nil, nil), never a NotFound gRPC
|
||||
// error (§7's found=false-not-error convention).
|
||||
missing, err := cl.TaskByID(ctx, task.ID+999999)
|
||||
if err != nil || missing != nil {
|
||||
t.Fatalf("TaskByID missing = (%+v, %v), want (nil, nil)", missing, err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("single_owner_sqlite", func(t *testing.T) {
|
||||
// The binary owns one task SQLite file (db_path + "-tasks.sqlite");
|
||||
// the consumer called only gRPC methods and never opened a store file.
|
||||
// The config anchor path itself must not be opened directly (no dual
|
||||
// store ownership, §12 step 8).
|
||||
taskFile := filepath.Join(dir, "archie.db-tasks.sqlite")
|
||||
if _, err := os.Stat(taskFile); err != nil {
|
||||
t.Fatalf("state store did not own %s: %v", taskFile, err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "archie.db")); err == nil {
|
||||
t.Fatalf("config anchor path was opened directly; dual store ownership violated §12 step 8")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestStateStoreRealProcessDeadline verifies the per-call deadline behaviour
|
||||
// the state store contract requires (§6 StoreTimeout): a consumer context that
|
||||
// is cancelled before the call completes must surface a deadline/cancel error
|
||||
// rather than hang or silently succeed.
|
||||
func TestStateStoreRealProcessDeadline(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("real-process smoke test builds and execs the binary; skip under -short")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
bin := buildBinary(t, dir)
|
||||
cfg := writeMinimalConfig(t, dir)
|
||||
st := startStateStoreProcess(t, bin, cfg)
|
||||
cl := dial(t, st.addr)
|
||||
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
cancel()
|
||||
_, err := cl.StatusCounts(ctx)
|
||||
if err == nil {
|
||||
t.Fatal("StatusCounts on a cancelled context returned nil; expected a deadline/cancel error")
|
||||
}
|
||||
if !errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("StatusCounts cancelled = %v, want context.Canceled or DeadlineExceeded", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStateStoreRealProcessRestartRecovery starts the process, writes a task,
|
||||
// stops it (SIGTERM), restarts it on the same config/db_path, and verifies the
|
||||
// task and its events were recovered from the on-disk SQLite -- the
|
||||
// restart/recovery path the runbook relies on when archie-state-store is
|
||||
// restarted as a service.
|
||||
func TestStateStoreRealProcessRestartRecovery(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("real-process smoke test builds and execs the binary; skip under -short")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
bin := buildBinary(t, dir)
|
||||
cfg := writeMinimalConfig(t, dir)
|
||||
|
||||
first := startStateStoreProcess(t, bin, cfg)
|
||||
s1 := dial(t, first.addr)
|
||||
task, err := s1.EnqueueChatTask(t.Context(), "acme", "widget", "survive restart", "body", "implement", "")
|
||||
if err != nil || task == nil {
|
||||
t.Fatalf("first EnqueueChatTask = %+v, %v", task, err)
|
||||
}
|
||||
if _, err := s1.InsertEvent(t.Context(), events.Event{Kind: "run_started", TaskID: task.ID}); err != nil {
|
||||
t.Fatalf("first InsertEvent: %v", err)
|
||||
}
|
||||
first.stop(t)
|
||||
|
||||
// Restart on the same config; the SQLite file persists on disk.
|
||||
second := startStateStoreProcess(t, bin, cfg)
|
||||
s2 := dial(t, second.addr)
|
||||
|
||||
got, err := s2.TaskByID(t.Context(), task.ID)
|
||||
if err != nil || got == nil || got.Title != "survive restart" {
|
||||
t.Fatalf("TaskByID after restart = %+v, %v", got, err)
|
||||
}
|
||||
if evs, err := s2.TaskEvents(t.Context(), task.ID); err != nil || len(evs) != 1 {
|
||||
t.Fatalf("TaskEvents after restart = %+v, %v", evs, err)
|
||||
}
|
||||
// Stale-running work reopened on a fresh process is recovered by
|
||||
// RecoverStale, which must return without error on the restarted store.
|
||||
if _, err := s2.RecoverStale(t.Context()); err != nil {
|
||||
t.Fatalf("RecoverStale after restart: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStateStoreRealProcessRemoteSurfaces exercises the capture/mapping/
|
||||
// binding surfaces the standalone process fronts (the daemon/webui surfaces
|
||||
// swapped to the adapter in .4.4/.4.5), proving one process serves the full
|
||||
// combined contract a consumer mixes.
|
||||
func TestStateStoreRealProcessRemoteSurfaces(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("real-process smoke test builds and execs the binary; skip under -short")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
bin := buildBinary(t, dir)
|
||||
cfg := writeMinimalConfig(t, dir)
|
||||
st := startStateStoreProcess(t, bin, cfg)
|
||||
|
||||
conn, err := grpc.NewClient("passthrough:///"+st.addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
|
||||
if err != nil {
|
||||
t.Fatalf("dial: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = conn.Close() })
|
||||
cl := staterpc.NewClient(conn)
|
||||
|
||||
ctx := t.Context()
|
||||
mappingID, err := cl.InsertMapping(ctx, mapping.Mapping{Name: "m1", Fields: []mapping.Field{{Name: "a", Path: "a", Type: mapping.TypeString}}})
|
||||
if err != nil || mappingID == 0 {
|
||||
t.Fatalf("InsertMapping = (%d, %v)", mappingID, err)
|
||||
}
|
||||
if got, err := cl.GetMapping(ctx, mappingID); err != nil || got == nil || got.Name != "m1" {
|
||||
t.Fatalf("GetMapping = %+v, %v", got, err)
|
||||
}
|
||||
bindingID, err := cl.InsertBinding(ctx, binding.Binding{
|
||||
Name: "b1", Matcher: binding.Matcher{Source: "sentry"}, MappingID: mappingID,
|
||||
Workflow: "implement", Secret: "0123456789abcdef0123456789abcdef",
|
||||
})
|
||||
if err != nil || bindingID == 0 {
|
||||
t.Fatalf("InsertBinding = (%d, %v)", bindingID, err)
|
||||
}
|
||||
// A binding starts as draft and must be edited (draft -> pending_approval)
|
||||
// then approved (pending_approval -> armed) before it is armed for a
|
||||
// source. Drive the state machine so ArmedBindingsForSource returns it.
|
||||
if err := cl.UpdateBinding(ctx, binding.Binding{
|
||||
ID: bindingID, Name: "b1", Matcher: binding.Matcher{Source: "sentry"},
|
||||
MappingID: mappingID, Workflow: "implement", Owner: "acme", Repo: "widget",
|
||||
Secret: "0123456789abcdef0123456789abcdef",
|
||||
}); err != nil {
|
||||
t.Fatalf("UpdateBinding: %v", err)
|
||||
}
|
||||
if err := cl.ApproveBinding(ctx, bindingID); err != nil {
|
||||
t.Fatalf("ApproveBinding: %v", err)
|
||||
}
|
||||
if armed, err := cl.ArmedBindingsForSource(ctx, "sentry"); err != nil || len(armed) != 1 || armed[0].ID != bindingID {
|
||||
t.Fatalf("ArmedBindingsForSource = %+v, %v", armed, err)
|
||||
}
|
||||
// Remote surfaces share error-sentinel fidelity on the wire too.
|
||||
missing := mapping.Mapping{ID: mappingID + 999999, Name: "x", Fields: []mapping.Field{{Name: "a", Path: "a", Type: mapping.TypeString}}}
|
||||
if err := cl.UpdateMapping(ctx, missing); !errors.Is(err, store.ErrMappingNotFound) {
|
||||
t.Fatalf("UpdateMapping missing = %v, want ErrMappingNotFound", err)
|
||||
}
|
||||
}
|
||||
@@ -61,3 +61,22 @@ archie-state-store -config ~/.config/archie/config.toml -listen 127.0.0.1:9090
|
||||
# or container-mode: bind the bridge gateway with a token
|
||||
archie-state-store -config ~/.config/archie/config.toml -listen 172.17.0.1:9090 -token <token>
|
||||
```
|
||||
|
||||
## Verifying the State Store
|
||||
|
||||
`archie-state-store` exposes an optional HTTP readiness surface when launched
|
||||
with `-ready-addr` (the systemd unit uses `127.0.0.1:9091`). Use it to confirm
|
||||
the process is up and the store is healthy before starting `archied`:
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9091/healthz # 200 = process up
|
||||
curl -s http://127.0.0.1:9091/health/detailed # state_db component -> 200, or 503 when degraded
|
||||
```
|
||||
|
||||
A running State Store owns exactly one task SQLite file (`<db_path>-tasks.sqlite`);
|
||||
`archied` and `archie-gateway` never open `archie.db` directly, so a second
|
||||
store process on the same `db_path`, or a consumer that opens the task file,
|
||||
indicates dual-store ownership (`docs/prds/state-store-contract.md` §12 step 7/8).
|
||||
The readiness probe is the single-writer check the runbook has against a
|
||||
degraded store: a `503` from `/health/detailed` means `archied` should be
|
||||
started only after the store is healthy.
|
||||
@@ -80,3 +80,29 @@ func TestOpenStateStoreAdapterRemoteFailsClosed(t *testing.T) {
|
||||
t.Fatal("non-loopback without a token should fail closed")
|
||||
}
|
||||
}
|
||||
|
||||
// TestStateStoreSingleOwnerComposition proves the single-owner SQLite
|
||||
// invariant at the composition root: both production consumers (the daemon
|
||||
// Run path and the gateway RunGateway) resolve their task-store surface from
|
||||
// openStateStoreAdapter, which REQUIRES [services.state].target and never
|
||||
// opens archie.db directly. After .4.6 the in-process store path is deleted
|
||||
// (docs/prds/state-store-contract.md §12 step 7), so an empty target is a
|
||||
// startup error rather than the old local default -- if it were not, the
|
||||
// daemon or gateway would open the single SQLite file behind the store
|
||||
// service's back, violating no-dual-store-ownership.
|
||||
func TestStateStoreSingleOwnerComposition(t *testing.T) {
|
||||
b := newBootstrap()
|
||||
b.secrets = &secret.Registry{}
|
||||
|
||||
// The daemon openStores path must not construct a local *store.Store.
|
||||
// openStores only resolves secrets and the forge client, then opens chat
|
||||
// sessions; the task store surface comes from the remote adapter, so no
|
||||
// archie.db task file is touched. This asserts openStateStoreAdapter (the
|
||||
// only task-store constructor) rejects an empty target.
|
||||
if err := b.openStateStoreAdapter(); err == nil {
|
||||
t.Fatal("empty services.state.target must error: the in-process store path is deleted (single owner = archie-state-store)")
|
||||
}
|
||||
if b.stateStore != nil {
|
||||
t.Fatalf("stateStore = %p, want nil (no local store opened)", b.stateStore)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
package staterpc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"google.golang.org/grpc/codes"
|
||||
"google.golang.org/grpc/status"
|
||||
|
||||
"github.com/samcharles93/archie-core/internal/store"
|
||||
)
|
||||
|
||||
// TestUnmapErrorPreservesContextIdentity pins the §6 deadline contract that
|
||||
// unmapError must rehydrate a gRPC Canceled / DeadlineExceeded status back to
|
||||
// the standard context sentinels. The agent's deadlineStore bounds each Store
|
||||
// call with context.WithTimeout and the workflow consumer checks
|
||||
// errors.Is(err, context.DeadlineExceeded) to mark a stage interrupted rather
|
||||
// than failed (workflow.go); if unmapError folded those into a sanitised
|
||||
// internal error, a merely interrupted stage would be parked as a failure.
|
||||
func TestUnmapErrorPreservesContextIdentity(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
err error
|
||||
want error
|
||||
}{
|
||||
{name: "cancelled", err: status.Error(codes.Canceled, context.Canceled.Error()), want: context.Canceled},
|
||||
{name: "deadline exceeded", err: status.Error(codes.DeadlineExceeded, context.DeadlineExceeded.Error()), want: context.DeadlineExceeded},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := unmapError(tt.err)
|
||||
if !errors.Is(got, tt.want) {
|
||||
t.Fatalf("unmapError(%q) = %v, want errors.Is(err, %v)", tt.err, got, tt.want)
|
||||
}
|
||||
if errors.Is(got, store.ErrStaleTransition) {
|
||||
t.Fatalf("unmapError(%q) = %v, must not map to a store sentinel", tt.err, got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestMapErrorPreservesContextIdentity pins the server side of the same §6
|
||||
// contract: if the store (or a server handler) returns a context error because
|
||||
// the caller's ctx expired mid-call, mapError must map it to the matching
|
||||
// gRPC code rather than sanitising it into codes.Internal. Otherwise the
|
||||
// client never sees the deadline identity and an interrupted stage is wrongly
|
||||
// treated as a failure.
|
||||
func TestMapErrorPreservesContextIdentity(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
err error
|
||||
want error
|
||||
}{
|
||||
{name: "cancelled", err: context.Canceled, want: context.Canceled},
|
||||
{name: "deadline exceeded", err: context.DeadlineExceeded, want: context.DeadlineExceeded},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
mapped := mapError(tt.err)
|
||||
if got := unmapError(mapped); !errors.Is(got, tt.want) {
|
||||
t.Fatalf("mapError/unmapError round-trip for %q = %v, want errors.Is(err, %v)", tt.err, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestUnmapErrorSentinelFidelity pins the §7 sentinel rehydration round-trip
|
||||
// for each store error code, so a consumer's errors.Is(err, store.ErrX) keeps
|
||||
// working across the wire.
|
||||
func TestUnmapErrorSentinelFidelity(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
store error
|
||||
rehyd error
|
||||
}{
|
||||
{name: "stale transition", store: store.ErrStaleTransition, rehyd: store.ErrStaleTransition},
|
||||
{name: "binding not found", store: store.ErrBindingNotFound, rehyd: store.ErrBindingNotFound},
|
||||
{name: "mapping not found", store: store.ErrMappingNotFound, rehyd: store.ErrMappingNotFound},
|
||||
{name: "binding overlap", store: store.ErrBindingOverlap, rehyd: store.ErrBindingOverlap},
|
||||
{name: "binding transition", store: store.ErrBindingTransition, rehyd: store.ErrBindingTransition},
|
||||
{name: "already dispatched", store: store.ErrAlreadyDispatched, rehyd: store.ErrAlreadyDispatched},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
// mapError converts the store sentinel to a gRPC status; unmapError
|
||||
// must recover the same sentinel. This is the round-trip the wire
|
||||
// contract guarantees (§7).
|
||||
mapped := mapError(tt.store)
|
||||
got := unmapError(mapped)
|
||||
if !errors.Is(got, tt.rehyd) {
|
||||
t.Fatalf("round-trip %q -> %q: got %v, want %v", tt.store, mapped, got, tt.rehyd)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestUnmapErrorDoesNotLeakInternalDetail pins the §7 sanitisation rule: an
|
||||
// infra/unknown error maps to codes.Internal and must not rehydrate to a store
|
||||
// sentinel, the public message must not carry the raw wrapped detail (SQL
|
||||
// paths, provider details), and a non-status error is returned unchanged.
|
||||
func TestUnmapErrorDoesNotLeakInternalDetail(t *testing.T) {
|
||||
const detail = "sql: path=/var/lib/archie/archie.db secret=abcdef"
|
||||
internal := mapError(errors.New(detail))
|
||||
got := unmapError(internal)
|
||||
if errors.Is(got, store.ErrStaleTransition) || errors.Is(got, store.ErrBindingNotFound) || errors.Is(got, store.ErrAlreadyDispatched) {
|
||||
t.Fatalf("unmapError(internal) = %v, must not map to a store sentinel", got)
|
||||
}
|
||||
if st, ok := status.FromError(got); !ok || st.Code() != codes.Internal {
|
||||
t.Fatalf("unmapError(internal) = %v, want a codes.Internal status", got)
|
||||
}
|
||||
if strings.Contains(got.Error(), detail) {
|
||||
t.Fatalf("unmapError leaked raw internal detail: %q", got.Error())
|
||||
}
|
||||
|
||||
// A raw (non-status) error is returned unchanged (not wrapped into a
|
||||
// status or a store sentinel).
|
||||
raw := errors.New("transport failure")
|
||||
got = unmapError(raw)
|
||||
if errors.Is(got, store.ErrStaleTransition) {
|
||||
t.Fatalf("unmapError(raw) = %v, must not map to a store sentinel", got)
|
||||
}
|
||||
if !errors.Is(got, raw) {
|
||||
t.Fatalf("unmapError(raw) = %v, want an error chain including the original", got)
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package staterpc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -251,10 +252,24 @@ const (
|
||||
// (which may contain SQL, provider detail, or secrets). Infra errors map to
|
||||
// codes.Internal with a sanitised message; the caller is expected to log the
|
||||
// full error server-side before calling mapError.
|
||||
//
|
||||
// A context cancellation/deadline raised by the store (or by a caller ctx that
|
||||
// expires mid-call, §6) must retain its identity, not be folded into
|
||||
// codes.Internal: gRPC-Go surfaces context.Canceled/DeadlineExceeded to the
|
||||
// client only for those exact codes, and the agent's workflow consumer
|
||||
// depends on errors.Is(err, context.DeadlineExceeded) to distinguish an
|
||||
// interrupted stage from a failed one. So a context error maps to its own
|
||||
// gRPC code (and unmapError rehydrates it back to the sentinel).
|
||||
func mapError(err error) error {
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
if errors.Is(err, context.Canceled) {
|
||||
return status.Error(codes.Canceled, context.Canceled.Error())
|
||||
}
|
||||
if errors.Is(err, context.DeadlineExceeded) {
|
||||
return status.Error(codes.DeadlineExceeded, context.DeadlineExceeded.Error())
|
||||
}
|
||||
switch {
|
||||
case errors.Is(err, store.ErrStaleTransition):
|
||||
return status.Error(codes.FailedPrecondition, msgStaleTransition)
|
||||
@@ -275,8 +290,17 @@ func mapError(err error) error {
|
||||
|
||||
// unmapError rehydrates a gRPC status error back to the store sentinel it
|
||||
// came from, so a caller's errors.Is(err, store.ErrX) keeps working across
|
||||
// the wire. A non-status error (e.g. a transport failure, a cancelled or
|
||||
// deadline-exceeded context) is returned unchanged.
|
||||
// the wire. A non-status error (e.g. a transport failure) is returned
|
||||
// unchanged.
|
||||
//
|
||||
// The deadline/cancel identity must survive too (§6): the agent's
|
||||
// deadlineStore bounds each Store call with context.WithTimeout, and the
|
||||
// workflow consumer checks errors.Is(err, context.DeadlineExceeded) to
|
||||
// decide whether a stage was interrupted by shutdown rather than failed
|
||||
// (workflow.go). gRPC-Go surfaces an expired or cancelled context as a
|
||||
// *status.Error whose code is DeadlineExceeded or Canceled, so we rehydrate
|
||||
// those back to the standard context sentinels -- otherwise the consumer
|
||||
// would (wrongly) park a task that was merely interrupted.
|
||||
func unmapError(err error) error {
|
||||
if err == nil {
|
||||
return nil
|
||||
@@ -286,6 +310,10 @@ func unmapError(err error) error {
|
||||
return err
|
||||
}
|
||||
switch st.Code() {
|
||||
case codes.Canceled:
|
||||
return context.Canceled
|
||||
case codes.DeadlineExceeded:
|
||||
return context.DeadlineExceeded
|
||||
case codes.FailedPrecondition:
|
||||
switch st.Message() {
|
||||
case msgStaleTransition:
|
||||
|
||||
Reference in new issue
Block a user