test(eda): pin playbook dispatch ledger docs and regression gaps
This commit is contained in:
1 parent
c9765d9afd
commit
a092e2198e
9 files changed
+167
-21
No files matched your search
@@ -207,7 +207,7 @@ structures found in legacy packages.
|
||||
changing this package or its callers.
|
||||
- The proto (`proto/state/v1/state.proto`, service `StateStoreService`,
|
||||
package `statev1` in `internal/contracts/state/v1/`) is one gRPC service
|
||||
fronting every ratified store contract (59 RPCs, including grant management
|
||||
fronting every ratified store contract (50 RPCs, including grant management
|
||||
and deprecated compatibility RPCs); the Go consumer facades
|
||||
stay narrow (`workflow.Store`, `store.TaskStore`, etc., all ≤8 methods
|
||||
except the `TaskStore` composite) via `staterpc.Client`'s multiple `var _`
|
||||
|
||||
@@ -168,11 +168,12 @@ treats as "another cycle raced us," not an error. This guarantee holds
|
||||
**across daemon restarts**, not just within one process's lifetime, because
|
||||
the ledger is a durable table, not in-memory state.
|
||||
|
||||
This ledger is also the direct precedent for the still-open EDA playbook
|
||||
engine idempotency question (`archie-core-t2db.17`, `docs/prds/eda-playbook-
|
||||
engine.md`'s execution-time gap 2) -- the same `(owner, event)`-keyed
|
||||
`INSERT OR IGNORE` shape almost certainly generalizes to playbook-run
|
||||
idempotency rather than needing a new mechanism invented from scratch.
|
||||
This ledger is also the direct precedent for the EDA playbook engine's
|
||||
execution-time idempotency, now landed (`archie-core-t2db.17` closed): the
|
||||
`playbook_dispatches` table in `internal/store/playbook_dispatches.go` copies
|
||||
`binding_dispatches`'s `INSERT OR IGNORE` conventions with a
|
||||
`(playbook_id, playbook_version, event_id, action_id)` key
|
||||
(`docs/prds/eda-playbook-engine.md` gap 2).
|
||||
|
||||
## What this document does not cover
|
||||
|
||||
|
||||
@@ -122,6 +122,7 @@ consumer-capability lines so no consumer acquires a surface it does not need
|
||||
| `BindingStore` (store) | — | 6 | `*Store` | `*staterpc.Client` |
|
||||
| `BindingDispatcher` (store) | — | 3 | `*Store` | `*staterpc.Client` |
|
||||
| `BindingTaskCreator` (store) | — | 1 | `*Store` | `*staterpc.Client` |
|
||||
| `PlaybookDispatcher` (store) | — | 2 | `*Store` | `*staterpc.Client` |
|
||||
|
||||
**Key property of the remote client:** `*staterpc.Client` is a *single* adapter that wraps one
|
||||
`StateStoreClient` and is asserted against **whichever narrow Go interfaces its caller needs**
|
||||
@@ -156,6 +157,7 @@ access through daemon-side narrow adapter interfaces.
|
||||
| `BindingStore` (6) | DAEMON (dashboard editor + dispatch loop) | not yet |
|
||||
| `BindingDispatcher` (3) | DAEMON (dispatch loop) | not yet |
|
||||
| `BindingTaskCreator` (1) | DAEMON (dispatch loop) | not yet |
|
||||
| `PlaybookDispatcher` (2) | DAEMON (EDA playbook dispatch loop) | **YES** (side-effecting-action idempotency ledger over gRPC) |
|
||||
|
||||
**Consequence for migration order:** the **agent's `workflow.Store` is already cross-process
|
||||
already** (via the NATS `storerpc` surrogate), so it is the first contract to move to gRPC. The
|
||||
@@ -292,6 +294,19 @@ condition, and the panel that collapsed them told operators task logging was swi
|
||||
it was not. The Go facade is `storecontract.TaskLogStore` (2 methods), carried by
|
||||
`*staterpc.Client` alongside the rest.
|
||||
|
||||
The playbook pair was added by `archie-core-t2db.17`'s implementation (see
|
||||
`docs/prds/eda-playbook-engine.md`, execution-time gap 2). It is administrative
|
||||
for the same reason as the config-snapshot and apply-status pairs, and
|
||||
deny-by-default in `authorizesTaskScopedCall` covers it without a new rule. The
|
||||
row is the structural `(playbook_id, playbook_version, event_id, action_id)`
|
||||
tuple, written **before** the action's side effect fires (the reverse of
|
||||
`binding_dispatches`), so a duplicate returns `store.ErrAlreadyDispatched` and
|
||||
the coordinator skips the invoke rather than repeating a non-revocable side
|
||||
effect. `DeletePlaybookDispatches` is the reclamation primitive for a playbook
|
||||
removed from the configured directory. The Go facade is
|
||||
`storecontract.PlaybookDispatcher` (2 methods), carried by `*staterpc.Client`
|
||||
alongside the rest.
|
||||
|
||||
### Domain types and `values.go` mapping
|
||||
|
||||
`values.go` maps proto ⇄ these types. The task model for the **agent** path is the domain-owned
|
||||
@@ -591,6 +606,7 @@ an explicit operator decision.
|
||||
var _ store.BindingStore = (*staterpc.Client)(nil)
|
||||
var _ store.BindingDispatcher = (*staterpc.Client)(nil)
|
||||
var _ store.BindingTaskCreator = (*staterpc.Client)(nil)
|
||||
var _ store.PlaybookDispatcher = (*staterpc.Client)(nil)
|
||||
// local side (in store), already present:
|
||||
var _ workflow.Store = (*Store)(nil)
|
||||
```
|
||||
|
||||
@@ -129,12 +129,13 @@ func TestStateStoreDepsServeTaskLogs(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestStateStoreDepsServePlaybookDispatcher proves the standalone State Store
|
||||
// process fronts the playbook dispatch ledger: the two playbook RPCs answer
|
||||
// codes.Unavailable for every call unless stateStoreDeps lifts the opened
|
||||
// *store.Store's PlaybookDispatcher surface onto Deps. This is the only
|
||||
// production server for that surface, so without this wiring the idempotency
|
||||
// ledger is unreachable.
|
||||
// TestStateStoreDepsServePlaybookDispatcher verifies stateStoreDeps lifts the
|
||||
// opened *store.Store's PlaybookDispatcher surface onto Deps, so the
|
||||
// standalone State Store has a server for the two playbook RPCs. It does not
|
||||
// dial those RPCs here -- it inspects the assembled Deps only. The nil check
|
||||
// is the load-bearing part: a boot without a store must leave
|
||||
// PlaybookDispatcher nil (the server then answers codes.Unavailable) rather
|
||||
// than fabricating a dispatcher.
|
||||
func TestStateStoreDepsServePlaybookDispatcher(t *testing.T) {
|
||||
st, err := store.Open(t.Context(), filepath.Join(t.TempDir(), "tasks.sqlite"))
|
||||
if err != nil {
|
||||
|
||||
@@ -2,6 +2,7 @@ package playbook
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"slices"
|
||||
"testing"
|
||||
@@ -15,6 +16,7 @@ import (
|
||||
// path does.
|
||||
type fakePlaybookLedger struct {
|
||||
records []playbookRecord
|
||||
err error
|
||||
}
|
||||
|
||||
type playbookRecord struct {
|
||||
@@ -25,6 +27,9 @@ type playbookRecord struct {
|
||||
}
|
||||
|
||||
func (f *fakePlaybookLedger) RecordPlaybookDispatch(ctx context.Context, playbookID, playbookVersion, eventID, actionID string) error {
|
||||
if f.err != nil {
|
||||
return f.err
|
||||
}
|
||||
key := playbookRecord{playbookID, playbookVersion, eventID, actionID}
|
||||
if slices.Contains(f.records, key) {
|
||||
return storecontract.ErrAlreadyDispatched
|
||||
@@ -100,3 +105,73 @@ func TestInvokeOnceRejectsEmptyKeyComponent(t *testing.T) {
|
||||
t.Fatalf("ledger records = %d, want 0 (validation must precede the ledger write)", len(ledger.records))
|
||||
}
|
||||
}
|
||||
|
||||
// TestInvokeOncePropagatesLedgerError: a non-duplicate ledger error must stop
|
||||
// the invoke. InvokeOnce's record-before-invoke contract only tolerates
|
||||
// ErrAlreadyDispatched; any other error is returned and the action must not
|
||||
// fire, or a transient ledger failure would swallow the error and run the
|
||||
// side effect anyway.
|
||||
func TestInvokeOncePropagatesLedgerError(t *testing.T) {
|
||||
ledgerErr := errors.New("ledger write failed")
|
||||
ledger := &fakePlaybookLedger{err: ledgerErr}
|
||||
decision := Decision{PlaybookID: "pb.yaml", Version: "v1", Workflow: "notify", ActionID: "notify"}
|
||||
input := DispatchInput{TaskID: "archie:acme/widget/7"}
|
||||
calls := 0
|
||||
action := func(ctx context.Context) error { calls++; return nil }
|
||||
|
||||
err := InvokeOnce(t.Context(), ledger, slog.New(slog.DiscardHandler), decision, input, action)
|
||||
if !errors.Is(err, ledgerErr) {
|
||||
t.Fatalf("InvokeOnce = %v, want the ledger error", err)
|
||||
}
|
||||
if calls != 0 {
|
||||
t.Fatalf("action invoked %d times, want 0 (a ledger error must stop the invoke)", calls)
|
||||
}
|
||||
}
|
||||
|
||||
// TestInvokeOnceKeysOnDispatchedDecision ties the coordinator to the real
|
||||
// Dispatch producer: the decision is not hand-built, it comes from a loaded
|
||||
// playbook's Store.Dispatch, so a Dispatch that stops populating ActionID or
|
||||
// ActionPosition fails here rather than silently collapsing ledger keys.
|
||||
func TestInvokeOnceKeysOnDispatchedDecision(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeFile(t, dir, "pb.yaml", `
|
||||
trigger:
|
||||
kind: bug
|
||||
actions:
|
||||
- position: workflow
|
||||
workflow: tdd
|
||||
id: notify
|
||||
`)
|
||||
store, err := Load(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("Load: %v", err)
|
||||
}
|
||||
decision, ok := store.Dispatch(DispatchInput{
|
||||
Labels: []string{"bug"},
|
||||
Kind: "bug",
|
||||
TaskID: "archie:acme/widget/7",
|
||||
Event: map[string]any{},
|
||||
})
|
||||
if !ok {
|
||||
t.Fatal("Dispatch matched nothing, want the tdd workflow")
|
||||
}
|
||||
if decision.ActionID != "notify" {
|
||||
t.Fatalf("Dispatch ActionID = %q, want notify (the ledger key depends on the producer populating it)", decision.ActionID)
|
||||
}
|
||||
if decision.ActionPosition != 1 {
|
||||
t.Fatalf("Dispatch ActionPosition = %d, want 1 (the ledger fallback depends on the producer populating it)", decision.ActionPosition)
|
||||
}
|
||||
|
||||
ledger := &fakePlaybookLedger{}
|
||||
input := DispatchInput{TaskID: "archie:acme/widget/7"}
|
||||
if err := InvokeOnce(t.Context(), ledger, slog.New(slog.DiscardHandler), decision, input, func(ctx context.Context) error { return nil }); err != nil {
|
||||
t.Fatalf("InvokeOnce: %v", err)
|
||||
}
|
||||
if len(ledger.records) != 1 {
|
||||
t.Fatalf("ledger records = %d, want 1", len(ledger.records))
|
||||
}
|
||||
want := playbookRecord{playbookID: decision.PlaybookID, playbookVersion: decision.Version, eventID: "archie:acme/widget/7", actionID: decision.ActionID}
|
||||
if got := ledger.records[0]; got != want {
|
||||
t.Fatalf("ledger record = %+v, want %+v", got, want)
|
||||
}
|
||||
}
|
||||
@@ -300,8 +300,12 @@ var (
|
||||
// ErrBindingTransition is returned when a binding state transition is
|
||||
// rejected by the draft -> pending_approval -> armed machine.
|
||||
ErrBindingTransition = errors.New("store: binding state transition rejected")
|
||||
// ErrAlreadyDispatched is returned when a capture already has a
|
||||
// dispatch ledger row (at-most-once dispatch).
|
||||
// ErrAlreadyDispatched is returned when a dedup ledger already holds the
|
||||
// dispatch key: binding_dispatches for the (binding_id, capture_id) pair,
|
||||
// or playbook_dispatches for the (playbook_id, playbook_version,
|
||||
// event_id, action_id) tuple. Both ledgers are at-most-once guarantees;
|
||||
// the sentinel is shared so a caller treats "already recorded" the same
|
||||
// way across the two surfaces.
|
||||
ErrAlreadyDispatched = errors.New("store: binding already dispatched for capture")
|
||||
// ErrMappingNotFound is returned when a mapping ID does not exist.
|
||||
ErrMappingNotFound = errors.New("store: mapping not found")
|
||||
|
||||
@@ -350,12 +350,20 @@ func TestStateStoreConformance(t *testing.T) {
|
||||
if err := pc.RecordPlaybookDispatch(ctx, "pb.yaml", "v1", "archie:acme/widget/7", "notify"); err != nil {
|
||||
t.Fatalf("RecordPlaybookDispatch: %v", err)
|
||||
}
|
||||
// A distinct event_id is a distinct row: the client -> proto ->
|
||||
// server mapping must carry event_id rather than collapse it into
|
||||
// the first tuple's key.
|
||||
// Three of the four PRIMARY KEY components are carried through the hop
|
||||
// with a distinguishing value here: a client/server mapping that drops
|
||||
// or transposes event_id, playbook_version, or action_id collapses
|
||||
// these distinct rows onto the first tuple and fails one of the three
|
||||
// below.
|
||||
if err := pc.RecordPlaybookDispatch(ctx, "pb.yaml", "v1", "archie:acme/widget/8", "notify"); err != nil {
|
||||
t.Fatalf("RecordPlaybookDispatch (distinct event_id) = %v, want success", err)
|
||||
}
|
||||
if err := pc.RecordPlaybookDispatch(ctx, "pb.yaml", "v2", "archie:acme/widget/7", "notify"); err != nil {
|
||||
t.Fatalf("RecordPlaybookDispatch (distinct playbook_version) = %v, want success", err)
|
||||
}
|
||||
if err := pc.RecordPlaybookDispatch(ctx, "pb.yaml", "v1", "archie:acme/widget/7", "build"); err != nil {
|
||||
t.Fatalf("RecordPlaybookDispatch (distinct action_id) = %v, want success", err)
|
||||
}
|
||||
err = pc.RecordPlaybookDispatch(ctx, "pb.yaml", "v1", "archie:acme/widget/7", "notify")
|
||||
if !errors.Is(err, store.ErrAlreadyDispatched) {
|
||||
t.Fatalf("RecordPlaybookDispatch dup = %v, want ErrAlreadyDispatched", err)
|
||||
|
||||
@@ -31,7 +31,7 @@ func grantsServer(t *testing.T, adminToken string) (grants *TaskGrants, dial fun
|
||||
grpc.ChainUnaryInterceptor(grants.UnaryInterceptor(adminToken)),
|
||||
grpc.ChainStreamInterceptor(grants.StreamInterceptor(adminToken)),
|
||||
)
|
||||
RegisterServer(server, Deps{Tasks: local, Captures: local, Bindings: local, BindingDispatcher: local, Grants: grants, ConfigSnapshots: local, ApplyStatus: local})
|
||||
RegisterServer(server, Deps{Tasks: local, Captures: local, Bindings: local, BindingDispatcher: local, PlaybookDispatcher: local, Grants: grants, ConfigSnapshots: local, ApplyStatus: local})
|
||||
go func() { _ = server.Serve(listener) }()
|
||||
t.Cleanup(func() { server.Stop(); _ = listener.Close() })
|
||||
|
||||
@@ -295,6 +295,45 @@ func TestOnlyAdminReportsApplyStatus(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestOnlyAdminOwnsThePlaybookLedger: the playbook dispatch ledger records
|
||||
// which side-effecting playbook action already fired, so a container's
|
||||
// task-scoped credential must not be able to write a "not dispatched" row
|
||||
// that would re-run a non-revocable side effect, nor to delete the ledger
|
||||
// and erase the skip.
|
||||
func TestOnlyAdminOwnsThePlaybookLedger(t *testing.T) {
|
||||
const adminToken = "daemon-admin-token"
|
||||
_, dial := grantsServer(t, adminToken)
|
||||
admin := dial(t, adminToken)
|
||||
ctx := t.Context()
|
||||
|
||||
task, err := admin.EnqueueChatTask(ctx, "acme", "widget", "a", "body", "implement", "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
workerToken, err := admin.RegisterTaskGrant(ctx, task.ID, time.Hour)
|
||||
if err != nil {
|
||||
t.Fatalf("RegisterTaskGrant: %v", err)
|
||||
}
|
||||
worker := dial(t, workerToken)
|
||||
|
||||
if err := admin.RecordPlaybookDispatch(ctx, "pb.yaml", "v1", "archie:acme/widget/7", "notify"); err != nil {
|
||||
t.Fatalf("admin RecordPlaybookDispatch: %v", err)
|
||||
}
|
||||
// A distinct event_id is deliberate: the ledger's own duplicate answer is
|
||||
// ErrAlreadyDispatched, so re-recording the admin's tuple would be non-nil
|
||||
// even for an authorized caller. A distinct tuple succeeds only if the
|
||||
// caller is authorized, which is exactly what must be refused here.
|
||||
if err := worker.RecordPlaybookDispatch(ctx, "pb.yaml", "v1", "archie:acme/widget/8", "notify"); err == nil {
|
||||
t.Fatal("a task grant must not authorize recording the playbook dispatch ledger")
|
||||
}
|
||||
if err := worker.DeletePlaybookDispatches(ctx, "pb.yaml"); err == nil {
|
||||
t.Fatal("a task grant must not authorize deleting the playbook dispatch ledger")
|
||||
}
|
||||
if err := admin.DeletePlaybookDispatches(ctx, "pb.yaml"); err != nil {
|
||||
t.Fatalf("admin DeletePlaybookDispatches: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyAdminCanRegisterOrRevokeTaskGrants(t *testing.T) {
|
||||
const adminToken = "daemon-admin-token"
|
||||
_, dial := grantsServer(t, adminToken)
|
||||
|
||||
@@ -54,9 +54,11 @@ CREATE TABLE IF NOT EXISTS binding_dispatches (
|
||||
// separate read.
|
||||
|
||||
// ErrAlreadyDispatched is returned by RecordDispatch when the (binding_id,
|
||||
// capture_id) pair already exists in binding_dispatches. The dedup ledger is
|
||||
// the at-most-once guarantee per capture; the caller is expected to surface
|
||||
// this to the dispatch loop rather than retry.
|
||||
// capture_id) pair already exists in binding_dispatches, and by
|
||||
// RecordPlaybookDispatch when the (playbook_id, playbook_version, event_id,
|
||||
// action_id) tuple already exists in playbook_dispatches. Each dedup ledger
|
||||
// is an at-most-once guarantee; the caller is expected to surface this to the
|
||||
// dispatch loop rather than retry.
|
||||
|
||||
// bindingTimeLayout follows captureTimeLayout's reasoning: fixed-width
|
||||
// RFC3339 so string and chronological order agree.
|
||||
|
||||
Reference in new issue
Block a user