feat(statestore): carry control-plane apply status on the store contract
Adds PutApplyStatus/ListApplyStatus so each process can report which version of a control-plane resource it is running, and the UI can read it back. docs/prds/control-plane-apply-status.md is the approved design. It goes on the State Store contract rather than the control plane, whose service is for editing a resource: a status report is not an edit. The config-snapshot pair is the precedent, and the deny-by-default arm of authorizesTaskScopedCall already refuses both new RPCs to a task grant, which TestOnlyAdminReportsApplyStatus pins. The row is keyed on (process, kind) and replaced in place, because a process re-stamps on an interval and nothing reads a history of re-stamps. An error leaves applied_version alone, so a rejected edit never erases the fact that an older version is live. The read is a list and an empty one is not an error, unlike the config snapshot's found=false singleton. Contract PRD inventory and RPC count follow in the same series. Relates to archie-core-pskb.
This commit is contained in:
1 parent
285474a552
commit
f38e1a30c8
15 files changed
+1048
-348
No files matched your search
@@ -1,13 +1,14 @@
|
||||
# Control-plane apply status
|
||||
|
||||
**Status:** Draft
|
||||
**Status:** Approved
|
||||
**Date:** 2026-09-21
|
||||
**Tracking:** `archie-core-pskb`, and `archie-core-nwa0` which depends on it
|
||||
|
||||
## Outcome
|
||||
|
||||
An operator who edits a control-plane resource can see which processes are
|
||||
running that version, which are still on an older one, and which rejected it.
|
||||
running that version, which are still on an older one, which rejected it, and
|
||||
which have stopped reporting.
|
||||
|
||||
`docs/prds/runtime-control-plane.md` requires this and does not say where the
|
||||
report goes. This decides that.
|
||||
@@ -33,6 +34,11 @@ One record per process and resource kind, replaced in place. It holds the
|
||||
process name, the resource kind, the version applied, the error if the process
|
||||
could not apply it, and the time it reported.
|
||||
|
||||
Each process re-stamps its records every 30 seconds. A record is stale once it
|
||||
is older than 90 seconds, and a stale record means the process that wrote it is
|
||||
no longer running. The report time is therefore a liveness signal, not only a
|
||||
write timestamp.
|
||||
|
||||
A process reports at the point it applies a resource, so the record says what
|
||||
happened rather than what was intended. Nothing probes.
|
||||
|
||||
@@ -75,28 +81,20 @@ and a failed write is logged rather than turned into a failed reload.
|
||||
The settings page shows, per resource, the stored version beside each process's
|
||||
applied version, error, and report time. A process on an older version of a
|
||||
restart-required resource is pending a restart. A process with an error is
|
||||
failed, with the error beside it.
|
||||
failed, with the error beside it. A process whose record is stale is unknown,
|
||||
never current.
|
||||
|
||||
## Open question: a record outlives the process that wrote it
|
||||
## A record outlives the process that wrote it
|
||||
|
||||
Records are replaced in place, so a process that applies a version and then
|
||||
stops leaves a record asserting that version indefinitely. The settings page
|
||||
would show a dead process as current. `docs/prds/status-health-surface.md`
|
||||
faced the same choice and stamped the start of a poll pass so a wedged poller
|
||||
reads as stale rather than healthy.
|
||||
stops would otherwise leave a record asserting that version indefinitely, and
|
||||
the settings page would show a dead process as current.
|
||||
`docs/prds/status-health-surface.md` faced the same choice and stamped the start
|
||||
of a poll pass so a wedged poller reads as stale rather than healthy.
|
||||
|
||||
This is not settled here, because the fix is a mechanism the tracking issue
|
||||
does not ask for. Three options:
|
||||
|
||||
1. Each process re-stamps its records on an interval, and a record that stops
|
||||
being re-stamped reads as unknown. Costs a recurring write per process.
|
||||
2. Records carry the instance identity generated at process start, and the UI
|
||||
reads a changed identity as a restart. Detects restarts, not deaths.
|
||||
3. Accept it. The page answers "what did each process apply" and not "is it
|
||||
still running", and says so.
|
||||
|
||||
Recommended: 1, because the question an operator brings to this page is whether
|
||||
their change is live now.
|
||||
Settled 2026-09-21: the re-stamp above. The alternatives considered were
|
||||
carrying the instance identity generated at process start, which detects a
|
||||
restart but not a death, and accepting the limitation with the page saying so.
|
||||
|
||||
## Verification
|
||||
|
||||
@@ -108,6 +106,7 @@ their change is live now.
|
||||
test binding writer to reader.
|
||||
- A task-scoped credential is refused by both RPCs.
|
||||
- A failed report does not fail the reload that produced it.
|
||||
- A record that stops being re-stamped renders as unknown rather than current.
|
||||
|
||||
## Not included
|
||||
|
||||
|
||||
@@ -245,6 +245,9 @@ func (b *boot) stateStoreDeps(grants *staterpc.TaskGrants) staterpc.Deps {
|
||||
if css, ok := b.st.(storecontract.ConfigSnapshotStore); ok {
|
||||
deps.ConfigSnapshots = css
|
||||
}
|
||||
if as, ok := b.st.(storecontract.ApplyStatusStore); ok {
|
||||
deps.ApplyStatus = as
|
||||
}
|
||||
return deps
|
||||
}
|
||||
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -53,6 +53,8 @@ const (
|
||||
StateStoreService_TokensByDay_FullMethodName = "/state.v1.StateStoreService/TokensByDay"
|
||||
StateStoreService_PutConfigSnapshot_FullMethodName = "/state.v1.StateStoreService/PutConfigSnapshot"
|
||||
StateStoreService_GetConfigSnapshot_FullMethodName = "/state.v1.StateStoreService/GetConfigSnapshot"
|
||||
StateStoreService_PutApplyStatus_FullMethodName = "/state.v1.StateStoreService/PutApplyStatus"
|
||||
StateStoreService_ListApplyStatus_FullMethodName = "/state.v1.StateStoreService/ListApplyStatus"
|
||||
StateStoreService_ReadTaskLog_FullMethodName = "/state.v1.StateStoreService/ReadTaskLog"
|
||||
StateStoreService_StreamTaskLogContent_FullMethodName = "/state.v1.StateStoreService/StreamTaskLogContent"
|
||||
StateStoreService_InsertCapture_FullMethodName = "/state.v1.StateStoreService/InsertCapture"
|
||||
@@ -133,6 +135,12 @@ type StateStoreServiceClient interface {
|
||||
// configuration page").
|
||||
PutConfigSnapshot(ctx context.Context, in *PutConfigSnapshotRequest, opts ...grpc.CallOption) (*PutConfigSnapshotResponse, error)
|
||||
GetConfigSnapshot(ctx context.Context, in *GetConfigSnapshotRequest, opts ...grpc.CallOption) (*GetConfigSnapshotResponse, error)
|
||||
// Apply status: which version of a control-plane resource each process is
|
||||
// running, and why it could not. Every process that applies a resource
|
||||
// publishes; the UI process reads (docs/prds/control-plane-apply-status.md).
|
||||
// Both are administrative.
|
||||
PutApplyStatus(ctx context.Context, in *PutApplyStatusRequest, opts ...grpc.CallOption) (*PutApplyStatusResponse, error)
|
||||
ListApplyStatus(ctx context.Context, in *ListApplyStatusRequest, opts ...grpc.CallOption) (*ListApplyStatusResponse, error)
|
||||
// Task log: one task attempt's persisted log. Task log files live in the
|
||||
// state directory this process owns, so the dashboard reads them over this
|
||||
// contract rather than opening them itself
|
||||
@@ -529,6 +537,26 @@ func (c *stateStoreServiceClient) GetConfigSnapshot(ctx context.Context, in *Get
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *stateStoreServiceClient) PutApplyStatus(ctx context.Context, in *PutApplyStatusRequest, opts ...grpc.CallOption) (*PutApplyStatusResponse, error) {
|
||||
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
|
||||
out := new(PutApplyStatusResponse)
|
||||
err := c.cc.Invoke(ctx, StateStoreService_PutApplyStatus_FullMethodName, in, out, cOpts...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *stateStoreServiceClient) ListApplyStatus(ctx context.Context, in *ListApplyStatusRequest, opts ...grpc.CallOption) (*ListApplyStatusResponse, error) {
|
||||
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
|
||||
out := new(ListApplyStatusResponse)
|
||||
err := c.cc.Invoke(ctx, StateStoreService_ListApplyStatus_FullMethodName, in, out, cOpts...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *stateStoreServiceClient) ReadTaskLog(ctx context.Context, in *ReadTaskLogRequest, opts ...grpc.CallOption) (*ReadTaskLogResponse, error) {
|
||||
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
|
||||
out := new(ReadTaskLogResponse)
|
||||
@@ -825,6 +853,12 @@ type StateStoreServiceServer interface {
|
||||
// configuration page").
|
||||
PutConfigSnapshot(context.Context, *PutConfigSnapshotRequest) (*PutConfigSnapshotResponse, error)
|
||||
GetConfigSnapshot(context.Context, *GetConfigSnapshotRequest) (*GetConfigSnapshotResponse, error)
|
||||
// Apply status: which version of a control-plane resource each process is
|
||||
// running, and why it could not. Every process that applies a resource
|
||||
// publishes; the UI process reads (docs/prds/control-plane-apply-status.md).
|
||||
// Both are administrative.
|
||||
PutApplyStatus(context.Context, *PutApplyStatusRequest) (*PutApplyStatusResponse, error)
|
||||
ListApplyStatus(context.Context, *ListApplyStatusRequest) (*ListApplyStatusResponse, error)
|
||||
// Task log: one task attempt's persisted log. Task log files live in the
|
||||
// state directory this process owns, so the dashboard reads them over this
|
||||
// contract rather than opening them itself
|
||||
@@ -983,6 +1017,12 @@ func (UnimplementedStateStoreServiceServer) PutConfigSnapshot(context.Context, *
|
||||
func (UnimplementedStateStoreServiceServer) GetConfigSnapshot(context.Context, *GetConfigSnapshotRequest) (*GetConfigSnapshotResponse, error) {
|
||||
return nil, status.Error(codes.Unimplemented, "method GetConfigSnapshot not implemented")
|
||||
}
|
||||
func (UnimplementedStateStoreServiceServer) PutApplyStatus(context.Context, *PutApplyStatusRequest) (*PutApplyStatusResponse, error) {
|
||||
return nil, status.Error(codes.Unimplemented, "method PutApplyStatus not implemented")
|
||||
}
|
||||
func (UnimplementedStateStoreServiceServer) ListApplyStatus(context.Context, *ListApplyStatusRequest) (*ListApplyStatusResponse, error) {
|
||||
return nil, status.Error(codes.Unimplemented, "method ListApplyStatus not implemented")
|
||||
}
|
||||
func (UnimplementedStateStoreServiceServer) ReadTaskLog(context.Context, *ReadTaskLogRequest) (*ReadTaskLogResponse, error) {
|
||||
return nil, status.Error(codes.Unimplemented, "method ReadTaskLog not implemented")
|
||||
}
|
||||
@@ -1679,6 +1719,42 @@ func _StateStoreService_GetConfigSnapshot_Handler(srv interface{}, ctx context.C
|
||||
return interceptor(ctx, in, info, handler)
|
||||
}
|
||||
|
||||
func _StateStoreService_PutApplyStatus_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
|
||||
in := new(PutApplyStatusRequest)
|
||||
if err := dec(in); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if interceptor == nil {
|
||||
return srv.(StateStoreServiceServer).PutApplyStatus(ctx, in)
|
||||
}
|
||||
info := &grpc.UnaryServerInfo{
|
||||
Server: srv,
|
||||
FullMethod: StateStoreService_PutApplyStatus_FullMethodName,
|
||||
}
|
||||
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
|
||||
return srv.(StateStoreServiceServer).PutApplyStatus(ctx, req.(*PutApplyStatusRequest))
|
||||
}
|
||||
return interceptor(ctx, in, info, handler)
|
||||
}
|
||||
|
||||
func _StateStoreService_ListApplyStatus_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
|
||||
in := new(ListApplyStatusRequest)
|
||||
if err := dec(in); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if interceptor == nil {
|
||||
return srv.(StateStoreServiceServer).ListApplyStatus(ctx, in)
|
||||
}
|
||||
info := &grpc.UnaryServerInfo{
|
||||
Server: srv,
|
||||
FullMethod: StateStoreService_ListApplyStatus_FullMethodName,
|
||||
}
|
||||
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
|
||||
return srv.(StateStoreServiceServer).ListApplyStatus(ctx, req.(*ListApplyStatusRequest))
|
||||
}
|
||||
return interceptor(ctx, in, info, handler)
|
||||
}
|
||||
|
||||
func _StateStoreService_ReadTaskLog_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
|
||||
in := new(ReadTaskLogRequest)
|
||||
if err := dec(in); err != nil {
|
||||
@@ -2179,6 +2255,14 @@ var StateStoreService_ServiceDesc = grpc.ServiceDesc{
|
||||
MethodName: "GetConfigSnapshot",
|
||||
Handler: _StateStoreService_GetConfigSnapshot_Handler,
|
||||
},
|
||||
{
|
||||
MethodName: "PutApplyStatus",
|
||||
Handler: _StateStoreService_PutApplyStatus_Handler,
|
||||
},
|
||||
{
|
||||
MethodName: "ListApplyStatus",
|
||||
Handler: _StateStoreService_ListApplyStatus_Handler,
|
||||
},
|
||||
{
|
||||
MethodName: "ReadTaskLog",
|
||||
Handler: _StateStoreService_ReadTaskLog_Handler,
|
||||
|
||||
@@ -109,6 +109,16 @@ type ConfigSnapshotStore interface {
|
||||
ConfigSnapshot(ctx context.Context) (ConfigSnapshot, bool, error)
|
||||
}
|
||||
|
||||
// ApplyStatusStore holds which version of each control-plane resource each
|
||||
// process is running. Same split as ConfigSnapshotStore and for the same
|
||||
// reason: the writers are the processes that apply a resource, the reader is
|
||||
// the UI, and a task-scoped credential must reach neither.
|
||||
// See docs/prds/control-plane-apply-status.md.
|
||||
type ApplyStatusStore interface {
|
||||
PutApplyStatus(ctx context.Context, status ApplyStatus) error
|
||||
ListApplyStatus(ctx context.Context) ([]ApplyStatus, error)
|
||||
}
|
||||
|
||||
// MappingStore persists payload field mappings (t2db.3). Deliberately
|
||||
// separate from TaskStore and CaptureStore for the same reason those are
|
||||
// split: the dashboard's mapping editor should only acquire the mapping
|
||||
@@ -214,6 +224,23 @@ type ConfigSnapshot struct {
|
||||
PublishedAt time.Time
|
||||
}
|
||||
|
||||
// ApplyStatus is one process's report about one control-plane resource kind.
|
||||
//
|
||||
// AppliedVersion is the version that process is running. Error is set when the
|
||||
// process could not apply what the store returned, and leaves AppliedVersion
|
||||
// at the last version it did apply, so a rejected edit never erases the fact
|
||||
// that an older one is live. ReportedAt is re-stamped on an interval: a record
|
||||
// older than the staleness window belongs to a process that has stopped
|
||||
// running, which is the only way this surface can tell a live process from a
|
||||
// dead one.
|
||||
type ApplyStatus struct {
|
||||
Process string `json:"process"`
|
||||
Kind string `json:"kind"`
|
||||
AppliedVersion int64 `json:"applied_version"`
|
||||
Error string `json:"error,omitempty"`
|
||||
ReportedAt time.Time `json:"reported_at"`
|
||||
}
|
||||
|
||||
// WorkflowStat is one row of the per-workflow metrics table.
|
||||
type WorkflowStat struct {
|
||||
Workflow string `json:"workflow"`
|
||||
|
||||
@@ -50,6 +50,7 @@ var (
|
||||
_ storecontract.BindingDispatcher = (*Client)(nil)
|
||||
_ storecontract.BindingTaskCreator = (*Client)(nil)
|
||||
_ storecontract.ConfigSnapshotStore = (*Client)(nil)
|
||||
_ storecontract.ApplyStatusStore = (*Client)(nil)
|
||||
_ identity.Repository = (*Client)(nil)
|
||||
)
|
||||
|
||||
@@ -289,6 +290,31 @@ func (c *Client) ConfigSnapshot(ctx context.Context) (storecontract.ConfigSnapsh
|
||||
return configSnapshotValue(reply.Snapshot), true, nil
|
||||
}
|
||||
|
||||
// PutApplyStatus reports what this process applied for one resource kind.
|
||||
// Administrative, like the config snapshot pair: a task-scoped grant cannot
|
||||
// reach it.
|
||||
func (c *Client) PutApplyStatus(ctx context.Context, status storecontract.ApplyStatus) error {
|
||||
_, err := c.client.PutApplyStatus(ctx, &pb.PutApplyStatusRequest{Status: applyStatusProto(status)})
|
||||
if err != nil {
|
||||
return unmapError(err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ListApplyStatus reads every process's report. An empty slice means nothing
|
||||
// has reported yet, which is not an error.
|
||||
func (c *Client) ListApplyStatus(ctx context.Context) ([]storecontract.ApplyStatus, error) {
|
||||
reply, err := c.client.ListApplyStatus(ctx, &pb.ListApplyStatusRequest{})
|
||||
if err != nil {
|
||||
return nil, unmapError(err)
|
||||
}
|
||||
statuses := make([]storecontract.ApplyStatus, 0, len(reply.Statuses))
|
||||
for _, status := range reply.Statuses {
|
||||
statuses = append(statuses, applyStatusValue(status))
|
||||
}
|
||||
return statuses, nil
|
||||
}
|
||||
|
||||
func (c *Client) ListCaptures(ctx context.Context, limit int) ([]storecontract.CapturedEvent, error) {
|
||||
stream, err := c.client.StreamCaptures(ctx, &pb.StreamCapturesRequest{Limit: int64(limit)})
|
||||
if err != nil {
|
||||
|
||||
@@ -29,6 +29,7 @@ type contract interface {
|
||||
store.BindingDispatcher
|
||||
store.BindingTaskCreator
|
||||
store.ConfigSnapshotStore
|
||||
store.ApplyStatusStore
|
||||
}
|
||||
|
||||
// taskLogContract is the task-log read group, driven separately because it is
|
||||
@@ -54,7 +55,7 @@ func remoteTaskStore(t *testing.T, local *store.Store, logs store.TaskLogStore)
|
||||
server := grpc.NewServer()
|
||||
RegisterServer(server, Deps{
|
||||
Tasks: local, Captures: local, Mappings: local, Bindings: local,
|
||||
BindingDispatcher: local, BindingTaskCreator: local, ConfigSnapshots: local,
|
||||
BindingDispatcher: local, BindingTaskCreator: local, ConfigSnapshots: local, ApplyStatus: local,
|
||||
TaskLogs: logs,
|
||||
})
|
||||
go func() { _ = server.Serve(listener) }()
|
||||
|
||||
@@ -31,6 +31,7 @@ import (
|
||||
var (
|
||||
errCaptureUnavailable = status.Error(codes.Unavailable, "capture store unavailable")
|
||||
errConfigSnapshotsUnavailable = status.Error(codes.Unavailable, "config snapshot store unavailable")
|
||||
errApplyStatusUnavailable = status.Error(codes.Unavailable, "apply status store unavailable")
|
||||
errMappingUnavailable = status.Error(codes.Unavailable, "mapping store unavailable")
|
||||
errBindingUnavailable = status.Error(codes.Unavailable, "binding store unavailable")
|
||||
errBindingDispatchUnavailable = status.Error(codes.Unavailable, "binding dispatcher unavailable")
|
||||
@@ -63,6 +64,7 @@ type Deps struct {
|
||||
Tasks storecontract.TaskStore
|
||||
Captures storecontract.CaptureStore
|
||||
ConfigSnapshots storecontract.ConfigSnapshotStore
|
||||
ApplyStatus storecontract.ApplyStatusStore
|
||||
Mappings storecontract.MappingStore
|
||||
Bindings storecontract.BindingStore
|
||||
BindingDispatcher storecontract.BindingDispatcher
|
||||
@@ -315,6 +317,40 @@ func (s *server) capture() (storecontract.CaptureStore, error) {
|
||||
return s.deps.Captures, nil
|
||||
}
|
||||
|
||||
func (s *server) applyStatus() (storecontract.ApplyStatusStore, error) {
|
||||
if s.deps.ApplyStatus == nil {
|
||||
return nil, errApplyStatusUnavailable
|
||||
}
|
||||
return s.deps.ApplyStatus, nil
|
||||
}
|
||||
|
||||
func (s *server) PutApplyStatus(ctx context.Context, r *pb.PutApplyStatusRequest) (*pb.PutApplyStatusResponse, error) {
|
||||
as, err := s.applyStatus()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := as.PutApplyStatus(ctx, applyStatusValue(r.Status)); err != nil {
|
||||
return nil, s.logErr("PutApplyStatus", err)
|
||||
}
|
||||
return &pb.PutApplyStatusResponse{}, nil
|
||||
}
|
||||
|
||||
func (s *server) ListApplyStatus(ctx context.Context, _ *pb.ListApplyStatusRequest) (*pb.ListApplyStatusResponse, error) {
|
||||
as, err := s.applyStatus()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
statuses, err := as.ListApplyStatus(ctx)
|
||||
if err != nil {
|
||||
return nil, s.logErr("ListApplyStatus", err)
|
||||
}
|
||||
out := make([]*pb.ApplyStatus, 0, len(statuses))
|
||||
for _, status := range statuses {
|
||||
out = append(out, applyStatusProto(status))
|
||||
}
|
||||
return &pb.ListApplyStatusResponse{Statuses: out}, nil
|
||||
}
|
||||
|
||||
func (s *server) configSnapshots() (storecontract.ConfigSnapshotStore, error) {
|
||||
if s.deps.ConfigSnapshots == nil {
|
||||
return nil, errConfigSnapshotsUnavailable
|
||||
|
||||
@@ -31,7 +31,7 @@ func grantsServer(t *testing.T, adminToken string) (grants *TaskGrants, dial fun
|
||||
grpc.ChainUnaryInterceptor(grants.UnaryInterceptor(adminToken)),
|
||||
grpc.ChainStreamInterceptor(grants.StreamInterceptor(adminToken)),
|
||||
)
|
||||
RegisterServer(server, Deps{Tasks: local, Captures: local, Bindings: local, BindingDispatcher: local, Grants: grants, ConfigSnapshots: local})
|
||||
RegisterServer(server, Deps{Tasks: local, Captures: local, Bindings: local, BindingDispatcher: local, Grants: grants, ConfigSnapshots: local, ApplyStatus: local})
|
||||
go func() { _ = server.Serve(listener) }()
|
||||
t.Cleanup(func() { server.Stop(); _ = listener.Close() })
|
||||
|
||||
@@ -259,6 +259,42 @@ func TestOnlyAdminPublishesTheConfigSnapshot(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestOnlyAdminReportsApplyStatus: apply status says which process is running
|
||||
// which settings version. A container's task-scoped credential must not be
|
||||
// able to claim a process applied something, nor to read the deployment's
|
||||
// process inventory.
|
||||
func TestOnlyAdminReportsApplyStatus(t *testing.T) {
|
||||
const adminToken = "daemon-admin-token"
|
||||
_, dial := grantsServer(t, adminToken)
|
||||
admin := dial(t, adminToken)
|
||||
ctx := t.Context()
|
||||
|
||||
task, err := admin.EnqueueChatTask(ctx, "acme", "widget", "a", "body", "implement", "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
workerToken, err := admin.RegisterTaskGrant(ctx, task.ID, time.Hour)
|
||||
if err != nil {
|
||||
t.Fatalf("RegisterTaskGrant: %v", err)
|
||||
}
|
||||
worker := dial(t, workerToken)
|
||||
|
||||
status := store.ApplyStatus{Process: "archied", Kind: "tool-settings", AppliedVersion: 2}
|
||||
if err := admin.PutApplyStatus(ctx, status); err != nil {
|
||||
t.Fatalf("admin PutApplyStatus: %v", err)
|
||||
}
|
||||
if err := worker.PutApplyStatus(ctx, status); err == nil {
|
||||
t.Fatal("a task grant must not authorize reporting apply status")
|
||||
}
|
||||
if _, err := worker.ListApplyStatus(ctx); err == nil {
|
||||
t.Fatal("a task grant must not authorize reading apply status")
|
||||
}
|
||||
got, err := admin.ListApplyStatus(ctx)
|
||||
if err != nil || len(got) != 1 {
|
||||
t.Fatalf("admin ListApplyStatus = (%d records, %v), want the reported one", len(got), err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyAdminCanRegisterOrRevokeTaskGrants(t *testing.T) {
|
||||
const adminToken = "daemon-admin-token"
|
||||
_, dial := grantsServer(t, adminToken)
|
||||
|
||||
@@ -401,6 +401,29 @@ func configSnapshotValue(snapshot *pb.ConfigSnapshot) storecontract.ConfigSnapsh
|
||||
}
|
||||
}
|
||||
|
||||
func applyStatusProto(status storecontract.ApplyStatus) *pb.ApplyStatus {
|
||||
return &pb.ApplyStatus{
|
||||
Process: status.Process,
|
||||
Kind: status.Kind,
|
||||
AppliedVersion: status.AppliedVersion,
|
||||
Error: status.Error,
|
||||
ReportedAt: timestamp(status.ReportedAt),
|
||||
}
|
||||
}
|
||||
|
||||
func applyStatusValue(status *pb.ApplyStatus) storecontract.ApplyStatus {
|
||||
if status == nil {
|
||||
return storecontract.ApplyStatus{}
|
||||
}
|
||||
return storecontract.ApplyStatus{
|
||||
Process: status.Process,
|
||||
Kind: status.Kind,
|
||||
AppliedVersion: status.AppliedVersion,
|
||||
Error: status.Error,
|
||||
ReportedAt: timeValue(status.ReportedAt),
|
||||
}
|
||||
}
|
||||
|
||||
// taskLogEntryProto and taskLogEntryValue mirror internal/logging.Entry, whose
|
||||
// Fields map crosses as a JSON object string exactly as events.Event.Data does
|
||||
// (see eventDataJSON above). The logging package owns that format end to end;
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
const applyStatusSchema = `
|
||||
CREATE TABLE IF NOT EXISTS apply_status (
|
||||
process TEXT NOT NULL,
|
||||
kind TEXT NOT NULL,
|
||||
applied_version INTEGER NOT NULL DEFAULT 0,
|
||||
error TEXT NOT NULL DEFAULT '',
|
||||
reported_at TEXT NOT NULL,
|
||||
PRIMARY KEY (process, kind)
|
||||
);
|
||||
`
|
||||
|
||||
// applyStatusTimeLayout matches the rest of the store's timestamp columns.
|
||||
const applyStatusTimeLayout = time.RFC3339
|
||||
|
||||
// PutApplyStatus records what one process applied for one resource kind. The
|
||||
// key is (process, kind) and a report replaces its own row: a process
|
||||
// re-stamps on an interval, and a history of re-stamps is not something any
|
||||
// reader wants.
|
||||
func (s *Store) PutApplyStatus(ctx context.Context, status ApplyStatus) error {
|
||||
reported := status.ReportedAt
|
||||
if reported.IsZero() {
|
||||
reported = time.Now()
|
||||
}
|
||||
_, err := s.db.ExecContext(ctx, `
|
||||
INSERT INTO apply_status (process, kind, applied_version, error, reported_at)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT(process, kind) DO UPDATE SET
|
||||
applied_version=excluded.applied_version,
|
||||
error=excluded.error,
|
||||
reported_at=excluded.reported_at`,
|
||||
status.Process, status.Kind, status.AppliedVersion, status.Error,
|
||||
reported.UTC().Format(applyStatusTimeLayout))
|
||||
if err != nil {
|
||||
return fmt.Errorf("store: put apply status: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ListApplyStatus returns every process's report. An empty slice is the
|
||||
// normal state of a store whose processes have not booted yet, not an error.
|
||||
func (s *Store) ListApplyStatus(ctx context.Context) ([]ApplyStatus, error) {
|
||||
rows, err := s.db.QueryContext(ctx, `
|
||||
SELECT process, kind, applied_version, error, reported_at
|
||||
FROM apply_status ORDER BY process, kind`)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("store: list apply status: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
statuses := []ApplyStatus{}
|
||||
for rows.Next() {
|
||||
var (
|
||||
status ApplyStatus
|
||||
reported string
|
||||
)
|
||||
if err := rows.Scan(&status.Process, &status.Kind, &status.AppliedVersion, &status.Error, &reported); err != nil {
|
||||
return nil, fmt.Errorf("store: scan apply status: %w", err)
|
||||
}
|
||||
status.ReportedAt, err = time.Parse(applyStatusTimeLayout, reported)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("store: parse apply status time: %w", err)
|
||||
}
|
||||
statuses = append(statuses, status)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("store: list apply status: %w", err)
|
||||
}
|
||||
return statuses, nil
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TestApplyStatusReplacesPerProcessAndKind pins the key: a process re-stamping
|
||||
// its report must overwrite its own row, not append one, or the settings page
|
||||
// accumulates a row per report and cannot say which is current.
|
||||
func TestApplyStatusReplacesPerProcessAndKind(t *testing.T) {
|
||||
s := openTest(t)
|
||||
first := time.Date(2026, 9, 21, 10, 0, 0, 0, time.UTC)
|
||||
|
||||
for _, status := range []ApplyStatus{
|
||||
{Process: "archied", Kind: "model-role-assignments", AppliedVersion: 3, ReportedAt: first},
|
||||
{Process: "archied", Kind: "repository-policies", AppliedVersion: 5, ReportedAt: first},
|
||||
{Process: "archie-gateway", Kind: "model-role-assignments", AppliedVersion: 2, ReportedAt: first},
|
||||
} {
|
||||
if err := s.PutApplyStatus(t.Context(), status); err != nil {
|
||||
t.Fatalf("PutApplyStatus(%s/%s): %v", status.Process, status.Kind, err)
|
||||
}
|
||||
}
|
||||
|
||||
restamp := first.Add(30 * time.Second)
|
||||
if err := s.PutApplyStatus(t.Context(), ApplyStatus{
|
||||
Process: "archied", Kind: "model-role-assignments", AppliedVersion: 4, ReportedAt: restamp,
|
||||
}); err != nil {
|
||||
t.Fatalf("re-stamp: %v", err)
|
||||
}
|
||||
|
||||
got, err := s.ListApplyStatus(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("ListApplyStatus: %v", err)
|
||||
}
|
||||
if len(got) != 3 {
|
||||
t.Fatalf("ListApplyStatus returned %d records, want 3: %+v", len(got), got)
|
||||
}
|
||||
for _, record := range got {
|
||||
if record.Process != "archied" || record.Kind != "model-role-assignments" {
|
||||
continue
|
||||
}
|
||||
if record.AppliedVersion != 4 {
|
||||
t.Errorf("AppliedVersion = %d, want the re-stamped 4", record.AppliedVersion)
|
||||
}
|
||||
if !record.ReportedAt.Equal(restamp) {
|
||||
t.Errorf("ReportedAt = %v, want the re-stamped %v", record.ReportedAt, restamp)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestApplyStatusKeepsAppliedVersionWhenReportingAnError pins that a rejected
|
||||
// edit does not erase the fact that an older version is still live.
|
||||
func TestApplyStatusKeepsAppliedVersionWhenReportingAnError(t *testing.T) {
|
||||
s := openTest(t)
|
||||
at := time.Date(2026, 9, 21, 10, 0, 0, 0, time.UTC)
|
||||
if err := s.PutApplyStatus(t.Context(), ApplyStatus{
|
||||
Process: "archied", Kind: "tool-settings", AppliedVersion: 7, ReportedAt: at,
|
||||
}); err != nil {
|
||||
t.Fatalf("PutApplyStatus: %v", err)
|
||||
}
|
||||
if err := s.PutApplyStatus(t.Context(), ApplyStatus{
|
||||
Process: "archied", Kind: "tool-settings", AppliedVersion: 7,
|
||||
Error: "validate database settings: bad policy", ReportedAt: at.Add(time.Minute),
|
||||
}); err != nil {
|
||||
t.Fatalf("PutApplyStatus with error: %v", err)
|
||||
}
|
||||
|
||||
got, err := s.ListApplyStatus(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("ListApplyStatus: %v", err)
|
||||
}
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("ListApplyStatus returned %d records, want 1", len(got))
|
||||
}
|
||||
if got[0].AppliedVersion != 7 || got[0].Error == "" {
|
||||
t.Fatalf("record = %+v, want version 7 still live alongside the error", got[0])
|
||||
}
|
||||
}
|
||||
|
||||
// TestListApplyStatusEmptyIsNotAnError pins the list convention: nothing has
|
||||
// reported yet is the normal state of a store whose processes have not booted,
|
||||
// and it is an empty slice rather than an error or a found=false.
|
||||
func TestListApplyStatusEmptyIsNotAnError(t *testing.T) {
|
||||
got, err := openTest(t).ListApplyStatus(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("ListApplyStatus on an empty store: %v", err)
|
||||
}
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("ListApplyStatus returned %d records, want none", len(got))
|
||||
}
|
||||
}
|
||||
@@ -28,12 +28,14 @@ type (
|
||||
TaskLogStore = storecontract.TaskLogStore
|
||||
CaptureStore = storecontract.CaptureStore
|
||||
ConfigSnapshotStore = storecontract.ConfigSnapshotStore
|
||||
ApplyStatusStore = storecontract.ApplyStatusStore
|
||||
MappingStore = storecontract.MappingStore
|
||||
BindingStore = storecontract.BindingStore
|
||||
BindingDispatcher = storecontract.BindingDispatcher
|
||||
BindingTaskCreator = storecontract.BindingTaskCreator
|
||||
CapturedEvent = storecontract.CapturedEvent
|
||||
ConfigSnapshot = storecontract.ConfigSnapshot
|
||||
ApplyStatus = storecontract.ApplyStatus
|
||||
WorkflowStat = storecontract.WorkflowStat
|
||||
StageStat = storecontract.StageStat
|
||||
DayTokens = storecontract.DayTokens
|
||||
|
||||
@@ -54,7 +54,7 @@ func Open(ctx context.Context, path string, opts ...OpenOption) (*Store, error)
|
||||
return nil, err
|
||||
}
|
||||
db.SetMaxOpenConns(1)
|
||||
if _, err := db.ExecContext(ctx, schema+eventsSchema+capturesSchema+mappingsSchema+bindingsSchema+configSnapshotSchema+resourcesSchema+identitiesSchema); err != nil {
|
||||
if _, err := db.ExecContext(ctx, schema+eventsSchema+capturesSchema+mappingsSchema+bindingsSchema+configSnapshotSchema+applyStatusSchema+resourcesSchema+identitiesSchema); err != nil {
|
||||
return nil, errors.Join(fmt.Errorf("store: init schema: %w", err), db.Close())
|
||||
}
|
||||
if err := migrateTasks(ctx, db); err != nil {
|
||||
|
||||
@@ -62,6 +62,13 @@ service StateStoreService {
|
||||
rpc PutConfigSnapshot(PutConfigSnapshotRequest) returns (PutConfigSnapshotResponse);
|
||||
rpc GetConfigSnapshot(GetConfigSnapshotRequest) returns (GetConfigSnapshotResponse);
|
||||
|
||||
// Apply status: which version of a control-plane resource each process is
|
||||
// running, and why it could not. Every process that applies a resource
|
||||
// publishes; the UI process reads (docs/prds/control-plane-apply-status.md).
|
||||
// Both are administrative.
|
||||
rpc PutApplyStatus(PutApplyStatusRequest) returns (PutApplyStatusResponse);
|
||||
rpc ListApplyStatus(ListApplyStatusRequest) returns (ListApplyStatusResponse);
|
||||
|
||||
// Task log: one task attempt's persisted log. Task log files live in the
|
||||
// state directory this process owns, so the dashboard reads them over this
|
||||
// contract rather than opening them itself
|
||||
@@ -450,6 +457,24 @@ message GetConfigSnapshotResponse {
|
||||
bool found = 2;
|
||||
}
|
||||
|
||||
// ApplyStatus is one process's report about one control-plane resource kind.
|
||||
// applied_version is the version that process is running; error is set when it
|
||||
// could not apply what the store returned, and leaves applied_version at the
|
||||
// last version it did apply. reported_at is re-stamped on an interval, so a
|
||||
// record older than the staleness window belongs to a process that has stopped
|
||||
// running (docs/prds/control-plane-apply-status.md).
|
||||
message ApplyStatus {
|
||||
string process = 1;
|
||||
string kind = 2;
|
||||
int64 applied_version = 3;
|
||||
string error = 4;
|
||||
google.protobuf.Timestamp reported_at = 5;
|
||||
}
|
||||
message PutApplyStatusRequest { ApplyStatus status = 1; }
|
||||
message PutApplyStatusResponse {}
|
||||
message ListApplyStatusRequest {}
|
||||
message ListApplyStatusResponse { repeated ApplyStatus statuses = 1; }
|
||||
|
||||
// TaskLogEntry mirrors internal/logging.Entry: one decoded log line. fields
|
||||
// carries every remaining key as a JSON object string, the same way Event
|
||||
// carries its data, so structured context survives the hop without this
|
||||
|
||||
Reference in new issue
Block a user