fix(archied): re-apply database-owned settings on SIGHUP reload
The reload callback re-resolved the file document and republished it as the running config. Every setting the control plane owns lives only in the database, so a reload reverted providers, model roles, repository policies, channel settings, scheduling policy, tool and plugin settings, container policies and workflow execution budgets to their file values until the process restarted, and logged "config reloaded" while doing it. Boot's layering is now one call, boot.runtimeConfig, and the reload makes the same call over the document it just resolved. Execution budgets arrive on a watch rather than in a resource query, so boot records the last settings it applied and the reload re-applies them from there. apply gains an error return, which the controller records exactly like a bad file: a control plane that cannot answer fails the reload and leaves the running config alone, because it already holds the values the file does not. The query is bounded at 10s so a State Store that stops answering surfaces as a failed reload rather than a SIGHUP that never returns. The callback also read the old config from b.d.Cfg, the same Holder as b.cfgHolder; reading it from the Holder boot owns drops the daemon from this path and makes it testable. Closes archie-core-ju85.
This commit is contained in:
7 files changed
+297
-63
No files matched your search
@@ -244,7 +244,7 @@ The application currently changes behavioural values through unrelated paths:
|
||||
| Active provider selection | `chatModelManager.SetActiveProvider` changes provider selection. | A conversation/session command changing runtime state, with the same distinction from provider-instance configuration. |
|
||||
| Persona selection | `PersonaRegistry.SetActive` mutates an in-memory per-session choice. | Agent-system session state. Persistence and audit requirements are owned by that domain, not the configuration loader. |
|
||||
| File and environment overlays | Applied while loading `internal/config`; defaults are filled by mutation in `finalize`. | Configuration-infrastructure input processing that produces separately typed candidates. Defaults and derivations remain visible in provenance. |
|
||||
| SIGHUP config reload (2026-08) | Re-resolves the file config, validates, then atomically republishes through `config.Holder`. A failed reload keeps the running config and records `last_error`/`last_error_at` in `/api/config`. It does not re-read the control plane, so database-owned settings revert to their file values until restart (`archie-core-ju85`). | Candidate promotion with last-known-good retention: the running snapshot is untouched on failure. Reload must re-apply the database layer. A bounded observation period and actor audit are not yet implemented. |
|
||||
| SIGHUP config reload (2026-08) | Re-resolves the file config, re-applies the database layer over it, validates, then atomically republishes through `config.Holder`. A failed reload, whether the file or the control plane caused it, keeps the running config and records `last_error`/`last_error_at` in `/api/config`. | Candidate promotion with last-known-good retention: the running snapshot is untouched on failure. A bounded observation period and actor audit are not yet implemented. |
|
||||
| Control-plane resource command (2026-09) | `ControlPlaneService.Command` validates a whole resource document against its owning feature, then writes it with an audit record (actor, source, request ID, expected version) under optimistic concurrency. Hosted on the State Store's gRPC server; the Web UI and messaging channels are its only clients. See `docs/prds/runtime-control-plane.md`. | Met for validate-stage-promote, actor audit and versioned history. Per-field health observation and validate-before-switch on live apply are not yet implemented. |
|
||||
|
||||
Any further mutation seam discovered during migration MUST be added here before
|
||||
@@ -307,11 +307,13 @@ deliberately. A reload that changes any non-allowlisted field logs
|
||||
|
||||
### SIGHUP reload
|
||||
|
||||
SIGHUP re-resolves the file config, validates, and atomically republishes
|
||||
through the shared Holder. A failed reload keeps the running config and records
|
||||
`last_error`/`last_error_at` in `/api/config` so the stale state is visible
|
||||
where the operator is looking. The three poll tickers re-read `PollInterval` per iteration and reset on change,
|
||||
so a reloaded interval genuinely takes effect.
|
||||
SIGHUP re-resolves the file config, re-applies the database layer over it
|
||||
(`boot.runtimeConfig`, the same call boot makes), validates, and atomically
|
||||
republishes through the shared Holder. A failed reload keeps the running config
|
||||
and records `last_error`/`last_error_at` in `/api/config` so the stale state is
|
||||
visible where the operator is looking. The three poll tickers re-read
|
||||
`PollInterval` per iteration and reset on change, so a reloaded interval
|
||||
genuinely takes effect.
|
||||
|
||||
### Settings writes go through the control plane
|
||||
|
||||
@@ -327,9 +329,13 @@ Two properties matter to anyone changing this area:
|
||||
layers database resources over it at boot (`boot.loadRuntimeConfig`). Values
|
||||
the daemon needs before it can reach the State Store — the database path,
|
||||
listen addresses, credentials — stay file-only by design.
|
||||
- **Reload does not re-read them.** SIGHUP republishes the file document alone,
|
||||
which reverts database-owned settings until restart. Tracked as
|
||||
`archie-core-ju85`; see the ledger above.
|
||||
- **Reload re-reads them.** SIGHUP resolves the file document and then runs the
|
||||
same `boot.runtimeConfig` layering boot does, so a reload cannot revert a
|
||||
database-owned setting to its file value. A control plane that cannot answer
|
||||
fails the reload rather than publishing the file's values over the running
|
||||
ones. Workflow execution budgets arrive on a watch rather than in a resource
|
||||
query, so boot records the last published settings and the reload re-applies
|
||||
them from there.
|
||||
|
||||
## Completion criteria
|
||||
|
||||
|
||||
@@ -113,7 +113,7 @@ posture is the opposite: **the daemon fails closed.**
|
||||
| --- | --- | --- |
|
||||
| State Store unreachable | `boot.loadRuntimeConfig` returns the dial error and `archied` exits non-zero | stderr: `runtime settings unavailable` |
|
||||
| A stored resource fails validation | Same: `validate database settings` wraps the error and the daemon exits | Same |
|
||||
| SIGHUP reload with a bad file | Running config is kept; the reload records `last_error` / `last_error_at` | `/api/config` → `reload.last_error` → dashboard banner |
|
||||
| SIGHUP reload with a bad file, or a control plane that cannot be read | Running config is kept; the reload records `last_error` / `last_error_at`. The reload re-applies the database layer, so it never publishes file values over the running database-owned ones | `/api/config` → `reload.last_error` → dashboard banner |
|
||||
|
||||
Failing closed is deliberate for settings the database owns: falling back to the
|
||||
file values would restore the second source of truth the control plane exists to
|
||||
|
||||
@@ -115,7 +115,11 @@ type boot struct {
|
||||
stateStoreGrants *staterpc.GrantIssuer
|
||||
stateStoreToken string
|
||||
controlPlane *controlplane.Client
|
||||
chatSessionStore gateway.SessionStore
|
||||
// executionSettings is the last workflow execution settings the control
|
||||
// plane published. They arrive on a watch rather than in the file
|
||||
// document, so reloadConfig re-applies them from here.
|
||||
executionSettings atomic.Pointer[workflow.ExecutionSettings]
|
||||
chatSessionStore gateway.SessionStore
|
||||
|
||||
catalog modelcatalog.Snapshot
|
||||
catalogModels []string
|
||||
@@ -1391,23 +1395,7 @@ func (b *boot) publishConfigSnapshot(ctx context.Context) {
|
||||
// /api/config.
|
||||
func (b *boot) wireConfigPublishing(ctx context.Context, cfgPath, overlayPath string) {
|
||||
log := b.log
|
||||
reloadController := newReloadController(b.loader, cfgPath, overlayPath, func(doc *configuration.Document) {
|
||||
// Boot merges catalog-discovered providers into cfg before the
|
||||
// Holders are seeded; publishing the raw reloaded document would
|
||||
// drop them from the running config even though the file is
|
||||
// unchanged. Re-apply the same merge (idempotent: a catalog that
|
||||
// failed to load merges to identity).
|
||||
applyModelCatalog(&doc.Config, b.catalog)
|
||||
old := b.d.Cfg.Get()
|
||||
b.currentProvenance.Store(&doc.Provenance)
|
||||
b.publishConfig(ctx, doc.Config)
|
||||
if fields := changedNonReloadableFields(old, doc.Config); len(fields) > 0 {
|
||||
log.Warn("config reloaded; some changes require a restart",
|
||||
"fields", fields, "paths", doc.Provenance.Paths())
|
||||
} else {
|
||||
log.Info("config reloaded", "paths", doc.Provenance.Paths())
|
||||
}
|
||||
})
|
||||
reloadController := newReloadController(b.loader, cfgPath, overlayPath, b.reloadConfig)
|
||||
b.lastReload = func() config.ReloadStatus {
|
||||
st := reloadController.Status()
|
||||
return st
|
||||
|
||||
@@ -3,25 +3,75 @@ package archied
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/samcharles93/archie-core/internal/config"
|
||||
"github.com/samcharles93/archie-core/internal/domain/workflow"
|
||||
"github.com/samcharles93/archie-core/internal/infrastructure/configuration"
|
||||
)
|
||||
|
||||
// runtimeConfig layers every database-owned setting over a file config and
|
||||
// validates the result. Boot and the SIGHUP reload both go through it: the
|
||||
// reload re-resolves the file document alone, so republishing that document
|
||||
// as it stands reverts each of these layers to its file value until the
|
||||
// process restarts (archie-core-ju85).
|
||||
func (b *boot) runtimeConfig(ctx context.Context, base config.Config) (config.Config, error) {
|
||||
cfg, err := b.controlPlane.RuntimeConfig(ctx, base)
|
||||
if err != nil {
|
||||
return config.Config{}, err
|
||||
}
|
||||
if settings := b.executionSettings.Load(); settings != nil {
|
||||
applyExecutionBudgets(&cfg, *settings)
|
||||
}
|
||||
if err := configuration.Validate(&cfg); err != nil {
|
||||
return config.Config{}, fmt.Errorf("validate database settings: %w", err)
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
func (b *boot) loadRuntimeConfig(ctx context.Context) error {
|
||||
cfg, err := b.controlPlane.RuntimeConfig(ctx, b.cfg)
|
||||
cfg, err := b.runtimeConfig(ctx, b.cfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := configuration.Validate(&cfg); err != nil {
|
||||
return fmt.Errorf("validate database settings: %w", err)
|
||||
}
|
||||
b.cfg = cfg
|
||||
b.cfgHolder.Set(cfg)
|
||||
return nil
|
||||
}
|
||||
|
||||
// reloadConfig publishes a freshly resolved file document as the running
|
||||
// config. The database layer is re-applied first, and a control plane that
|
||||
// cannot answer aborts the reload: the running config already holds those
|
||||
// values, so publishing the file's over them is the worse outcome.
|
||||
func (b *boot) reloadConfig(ctx context.Context, doc *configuration.Document) error {
|
||||
// Boot merges catalog-discovered providers into cfg before the
|
||||
// Holders are seeded; publishing the raw reloaded document would
|
||||
// drop them from the running config even though the file is
|
||||
// unchanged. Re-apply the same merge (idempotent: a catalog that
|
||||
// failed to load merges to identity).
|
||||
applyModelCatalog(&doc.Config, b.catalog)
|
||||
// Bounded: this runs on the signal loop, which handles nothing else
|
||||
// while it waits, and a State Store that has stopped answering must
|
||||
// surface as a failed reload rather than a SIGHUP that never returns.
|
||||
queryCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
cfg, err := b.runtimeConfig(queryCtx, doc.Config)
|
||||
if err != nil {
|
||||
return fmt.Errorf("runtime settings unavailable: %w", err)
|
||||
}
|
||||
doc.Config = cfg
|
||||
old := b.cfgHolder.Get()
|
||||
b.currentProvenance.Store(&doc.Provenance)
|
||||
b.publishConfig(ctx, cfg)
|
||||
if fields := changedNonReloadableFields(old, cfg); len(fields) > 0 {
|
||||
b.log.Warn("config reloaded; some changes require a restart",
|
||||
"fields", fields, "paths", doc.Provenance.Paths())
|
||||
} else {
|
||||
b.log.Info("config reloaded", "paths", doc.Provenance.Paths())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b *boot) startWorkflowExecutionSettings(ctx context.Context) error {
|
||||
settings, version, err := b.controlPlane.WorkflowExecutionSettings(ctx)
|
||||
if err != nil {
|
||||
@@ -44,18 +94,22 @@ func (b *boot) startWorkflowExecutionSettings(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// applyWorkflowExecutionSettings records the settings as well as applying
|
||||
// them: they arrive on a watch rather than in the file document, so a reload
|
||||
// has nowhere else to read them back from.
|
||||
func (b *boot) applyWorkflowExecutionSettings(settings workflow.ExecutionSettings, version int64) {
|
||||
b.executionSettings.Store(&settings)
|
||||
cfg := b.cfgHolder.Get().Clone()
|
||||
cfg.Budgets = executionBudgets(settings)
|
||||
for i := range cfg.Identities {
|
||||
cfg.Identities[i].Budgets = cfg.Budgets
|
||||
}
|
||||
applyExecutionBudgets(&cfg, settings)
|
||||
b.cfgHolder.Set(cfg)
|
||||
b.log.Info("workflow execution settings applied", "version", version)
|
||||
}
|
||||
|
||||
func executionBudgets(settings workflow.ExecutionSettings) config.Budgets {
|
||||
return config.Budgets{
|
||||
func applyExecutionBudgets(cfg *config.Config, settings workflow.ExecutionSettings) {
|
||||
cfg.Budgets = config.Budgets{
|
||||
MaxSteps: settings.MaxModelToolSteps, WallClock: config.Duration(settings.MaxRuntime), GateMaxFailures: settings.MaxConsecutiveGateFailures,
|
||||
}
|
||||
for i := range cfg.Identities {
|
||||
cfg.Identities[i].Budgets = cfg.Budgets
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,157 @@
|
||||
package archied
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"google.golang.org/grpc"
|
||||
|
||||
"github.com/samcharles93/archie-core/internal/app/controlplane"
|
||||
"github.com/samcharles93/archie-core/internal/config"
|
||||
pb "github.com/samcharles93/archie-core/internal/contracts/controlplane/v1"
|
||||
"github.com/samcharles93/archie-core/internal/domain/workflow"
|
||||
"github.com/samcharles93/archie-core/internal/infrastructure/configuration"
|
||||
)
|
||||
|
||||
// controlPlaneStub answers Query from a fixed resource map. queryErr, when
|
||||
// set, fails every Query so a test can drive the unreachable-State-Store
|
||||
// path.
|
||||
type controlPlaneStub struct {
|
||||
values map[string]any
|
||||
queryErr error
|
||||
}
|
||||
|
||||
func (*controlPlaneStub) Catalog(context.Context, *pb.CatalogRequest, ...grpc.CallOption) (*pb.CatalogResponse, error) {
|
||||
return &pb.CatalogResponse{}, nil
|
||||
}
|
||||
|
||||
func (c *controlPlaneStub) Query(_ context.Context, request *pb.QueryRequest, _ ...grpc.CallOption) (*pb.QueryResponse, error) {
|
||||
if c.queryErr != nil {
|
||||
return nil, c.queryErr
|
||||
}
|
||||
value, err := json.Marshal(c.values[request.Kind])
|
||||
return &pb.QueryResponse{Resource: &pb.Resource{Kind: request.Kind, Version: 3, ValueJson: value}}, err
|
||||
}
|
||||
|
||||
func (*controlPlaneStub) History(context.Context, *pb.HistoryRequest, ...grpc.CallOption) (*pb.HistoryResponse, error) {
|
||||
panic("unexpected History")
|
||||
}
|
||||
|
||||
func (*controlPlaneStub) Command(context.Context, *pb.CommandRequest, ...grpc.CallOption) (*pb.CommandResponse, error) {
|
||||
panic("unexpected Command")
|
||||
}
|
||||
|
||||
func (*controlPlaneStub) Watch(context.Context, *pb.WatchRequest, ...grpc.CallOption) (grpc.ServerStreamingClient[pb.WatchResponse], error) {
|
||||
panic("unexpected Watch")
|
||||
}
|
||||
|
||||
// databaseOwnedResources is a control-plane projection whose every value
|
||||
// differs from fileConfig below, so any field that comes back with the file
|
||||
// value after a reload is a field the reload reverted.
|
||||
func databaseOwnedResources() map[string]any {
|
||||
return map[string]any{
|
||||
controlplane.ProviderSettingsKind: map[string]any{"main": map[string]any{"class": "openai", "base_url": "https://db.example.com"}},
|
||||
controlplane.ModelRoleAssignmentsKind: map[string]string{"builder": "main/db-model"},
|
||||
controlplane.RepositoryPoliciesKind: []config.Repo{{Owner: "acme", Name: "from-database"}},
|
||||
controlplane.ChannelSettingsKind: map[string]any{"operator": "database", "max_steps": 20, "models": []string{"main/db-model"}, "email": map[string]any{}, "webhook": map[string]any{}, "telegram": map[string]any{"allowed_user_ids": []int64{42}}, "rate_limit": map[string]any{}, "workspace": "/workspace"},
|
||||
controlplane.SchedulingPolicyKind: map[string]any{"poll_interval": "2m", "max_retries": 7, "dispatch": map[string]any{"trigger": "assignee"}},
|
||||
controlplane.ToolSettingsKind: map[string]any{"mcp_servers": []map[string]any{}, "policy": map[string]any{}, "web_fetch": map[string]any{}, "minimax": map[string]any{}},
|
||||
controlplane.PluginSettingsKind: map[string]any{"plugin_dir": "/db/plugins", "module_dir": "/db/modules", "secret_engine_dir": "/db/secrets", "skills_dir": "/db/skills"},
|
||||
controlplane.ContainerRuntimePoliciesKind: config.ContainerConfig{Image: "archie:from-database", PullPolicy: "missing"},
|
||||
}
|
||||
}
|
||||
|
||||
// fileConfig is what the loader produces from config.toml alone: the layer a
|
||||
// SIGHUP reload re-resolves, holding none of the database's values.
|
||||
func fileConfig() config.Config {
|
||||
return config.Config{
|
||||
BotUser: "widget",
|
||||
Forge: config.Forge{Type: "github"},
|
||||
Providers: map[string]config.Provider{"file": {Class: "openai"}},
|
||||
Models: map[string]string{"builder": "file/model"},
|
||||
Repos: []config.Repo{{Owner: "acme", Name: "from-file"}},
|
||||
Chat: config.ChatConfig{Operator: "file"},
|
||||
PollInterval: config.Duration(time.Minute),
|
||||
MaxRetries: 1,
|
||||
PluginDir: "/file/plugins",
|
||||
Containers: config.ContainerConfig{Image: "archie:from-file", PullPolicy: "missing"},
|
||||
Budgets: config.Budgets{MaxSteps: 1, WallClock: config.Duration(time.Minute)},
|
||||
}
|
||||
}
|
||||
|
||||
func newReloadBoot(t *testing.T, stub *controlPlaneStub) *boot {
|
||||
t.Helper()
|
||||
b := &boot{
|
||||
cfg: fileConfig(),
|
||||
log: slog.New(slog.DiscardHandler),
|
||||
controlPlane: controlplane.NewRPCClient(stub),
|
||||
}
|
||||
b.cfgHolder = config.NewHolder(b.cfg)
|
||||
return b
|
||||
}
|
||||
|
||||
// TestReloadConfigKeepsDatabaseOwnedSettings is archie-core-ju85. The SIGHUP
|
||||
// callback re-resolves the file document alone; without re-applying the
|
||||
// control plane over it, every database-owned setting reverts to its file
|
||||
// value until the process restarts, and the reload still logs success.
|
||||
func TestReloadConfigKeepsDatabaseOwnedSettings(t *testing.T) {
|
||||
b := newReloadBoot(t, &controlPlaneStub{values: databaseOwnedResources()})
|
||||
if err := b.loadRuntimeConfig(t.Context()); err != nil {
|
||||
t.Fatalf("loadRuntimeConfig: %v", err)
|
||||
}
|
||||
b.applyWorkflowExecutionSettings(workflow.ExecutionSettings{
|
||||
MaxModelToolSteps: 99, MaxRuntime: time.Hour, MaxConsecutiveGateFailures: 5,
|
||||
}, 4)
|
||||
|
||||
if err := b.reloadConfig(t.Context(), &configuration.Document{Config: fileConfig()}); err != nil {
|
||||
t.Fatalf("reloadConfig: %v", err)
|
||||
}
|
||||
|
||||
got := b.cfgHolder.Get()
|
||||
for _, tt := range []struct {
|
||||
field string
|
||||
got any
|
||||
want any
|
||||
}{
|
||||
{"Providers", got.Providers["main"].BaseURL, "https://db.example.com"},
|
||||
{"Models", got.Models["builder"], "main/db-model"},
|
||||
{"Repos", got.Repos[0].Name, "from-database"},
|
||||
{"Chat", got.Chat.Operator, "database"},
|
||||
{"PollInterval", time.Duration(got.PollInterval), 2 * time.Minute},
|
||||
{"MaxRetries", got.MaxRetries, 7},
|
||||
{"PluginDir", got.PluginDir, "/db/plugins"},
|
||||
{"Containers", got.Containers.Image, "archie:from-database"},
|
||||
{"Budgets", got.Budgets.MaxSteps, 99},
|
||||
} {
|
||||
if tt.got != tt.want {
|
||||
t.Errorf("%s reverted to its file value after reload: got %v, want %v", tt.field, tt.got, tt.want)
|
||||
}
|
||||
}
|
||||
if got.BotUser != "widget" {
|
||||
t.Errorf("file-owned BotUser = %q, want the reloaded file value", got.BotUser)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReloadConfigFailsWhenControlPlaneUnreachable pins the direction of the
|
||||
// failure: the running config already holds the database's values, so a
|
||||
// control plane that cannot answer must abort the reload rather than publish
|
||||
// the file's values over them.
|
||||
func TestReloadConfigFailsWhenControlPlaneUnreachable(t *testing.T) {
|
||||
stub := &controlPlaneStub{values: databaseOwnedResources()}
|
||||
b := newReloadBoot(t, stub)
|
||||
if err := b.loadRuntimeConfig(t.Context()); err != nil {
|
||||
t.Fatalf("loadRuntimeConfig: %v", err)
|
||||
}
|
||||
|
||||
stub.queryErr = context.DeadlineExceeded
|
||||
err := b.reloadConfig(t.Context(), &configuration.Document{Config: fileConfig()})
|
||||
if err == nil {
|
||||
t.Fatal("reloadConfig with an unreachable control plane: expected an error")
|
||||
}
|
||||
if got := b.cfgHolder.Get(); got.Models["builder"] != "main/db-model" {
|
||||
t.Fatalf("running config was replaced on the error path: Models = %v", got.Models)
|
||||
}
|
||||
}
|
||||
@@ -14,10 +14,11 @@ import (
|
||||
|
||||
// ReloadController re-runs the configuration load on demand and
|
||||
// publishes the result through apply. A failed reload never publishes:
|
||||
// apply is called only with a Document that passed validation, and the
|
||||
// running config is left untouched on error. That "Set is not called on
|
||||
// the error path" property is the safety core of the reload feature --
|
||||
// a bad file must leave the daemon running on the previous config.
|
||||
// apply is called only with a Document that passed validation, and apply
|
||||
// itself reports failure rather than publishing a half-built config. That
|
||||
// "Set is not called on the error path" property is the safety core of the
|
||||
// reload feature -- a bad file, or a database layer that cannot be read,
|
||||
// must leave the daemon running on the previous config.
|
||||
type ReloadController struct {
|
||||
loader *configuration.Loader
|
||||
base string
|
||||
@@ -26,18 +27,14 @@ type ReloadController struct {
|
||||
// apply receives a freshly loaded, validated Document. It is the
|
||||
// composition root's job to wire this to whatever owns the running
|
||||
// config (the daemon's Holder, the dashboard's provenance). Must be
|
||||
// non-nil; newReloadController requires it.
|
||||
apply func(*configuration.Document)
|
||||
|
||||
// overlayValues, when non-nil, returns the runtime config overlay
|
||||
// (dotted keys -> typed values) to layer over the file config before
|
||||
// publishing. A failed overlay read or validation aborts the reload
|
||||
// exactly like a bad file.
|
||||
// non-nil; newReloadController requires it. An error from apply fails
|
||||
// the reload and leaves the running config alone.
|
||||
apply func(context.Context, *configuration.Document) error
|
||||
|
||||
status atomic.Pointer[config.ReloadStatus]
|
||||
}
|
||||
|
||||
func newReloadController(loader *configuration.Loader, base, overlay string, apply func(*configuration.Document)) *ReloadController {
|
||||
func newReloadController(loader *configuration.Loader, base, overlay string, apply func(context.Context, *configuration.Document) error) *ReloadController {
|
||||
c := &ReloadController{loader: loader, base: base, overlay: overlay, apply: apply}
|
||||
c.status.Store(&config.ReloadStatus{})
|
||||
return c
|
||||
@@ -45,10 +42,13 @@ func newReloadController(loader *configuration.Loader, base, overlay string, app
|
||||
|
||||
// Reload re-runs the configuration load. On validation failure it
|
||||
// records the error in Status and returns it; apply is not called and
|
||||
// the running configuration is unchanged. On success it calls apply
|
||||
// with the fresh Document and records the reload time.
|
||||
func (c *ReloadController) Reload() error {
|
||||
// the running configuration is unchanged. An error from apply is
|
||||
// recorded the same way. On success it records the reload time.
|
||||
func (c *ReloadController) Reload(ctx context.Context) error {
|
||||
doc, err := c.loader.Resolve(c.base, c.overlay)
|
||||
if err == nil {
|
||||
err = c.apply(ctx, doc)
|
||||
}
|
||||
if err != nil {
|
||||
c.status.Store(&config.ReloadStatus{
|
||||
LastError: err.Error(),
|
||||
@@ -56,7 +56,6 @@ func (c *ReloadController) Reload() error {
|
||||
})
|
||||
return err
|
||||
}
|
||||
c.apply(doc)
|
||||
c.status.Store(&config.ReloadStatus{
|
||||
LastReloadAt: time.Now().UTC().Format(time.RFC3339),
|
||||
})
|
||||
@@ -80,7 +79,7 @@ func reloadLoop(ctx context.Context, ch <-chan os.Signal, c *ReloadController, l
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ch:
|
||||
if err := c.Reload(); err != nil {
|
||||
if err := c.Reload(ctx); err != nil {
|
||||
log.Error("config reload failed; keeping running config", "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ package archied
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -40,11 +41,12 @@ func TestReloadAppliesAndPublishes(t *testing.T) {
|
||||
|
||||
loader := configuration.New(slog.New(slog.DiscardHandler))
|
||||
var published config.Config
|
||||
controller := newReloadController(loader, path, "", func(doc *configuration.Document) {
|
||||
controller := newReloadController(loader, path, "", func(_ context.Context, doc *configuration.Document) error {
|
||||
published = doc.Config
|
||||
return nil
|
||||
})
|
||||
|
||||
if err := controller.Reload(); err != nil {
|
||||
if err := controller.Reload(t.Context()); err != nil {
|
||||
t.Fatalf("first reload: %v", err)
|
||||
}
|
||||
if published.BotUser != "first" {
|
||||
@@ -52,7 +54,7 @@ func TestReloadAppliesAndPublishes(t *testing.T) {
|
||||
}
|
||||
|
||||
writeConfig(t, path, minimalConfigTOML("second"))
|
||||
if err := controller.Reload(); err != nil {
|
||||
if err := controller.Reload(t.Context()); err != nil {
|
||||
t.Fatalf("second reload: %v", err)
|
||||
}
|
||||
if published.BotUser != "second" {
|
||||
@@ -74,9 +76,9 @@ func TestReloadValidationFailureDoesNotApply(t *testing.T) {
|
||||
|
||||
loader := configuration.New(slog.New(slog.DiscardHandler))
|
||||
applied := 0
|
||||
controller := newReloadController(loader, path, "", func(*configuration.Document) { applied++ })
|
||||
controller := newReloadController(loader, path, "", func(context.Context, *configuration.Document) error { applied++; return nil })
|
||||
|
||||
if err := controller.Reload(); err != nil {
|
||||
if err := controller.Reload(t.Context()); err != nil {
|
||||
t.Fatalf("first reload: %v", err)
|
||||
}
|
||||
if applied != 1 {
|
||||
@@ -84,7 +86,7 @@ func TestReloadValidationFailureDoesNotApply(t *testing.T) {
|
||||
}
|
||||
|
||||
writeConfig(t, path, invalidConfigTOML())
|
||||
err := controller.Reload()
|
||||
err := controller.Reload(t.Context())
|
||||
if err == nil {
|
||||
t.Fatal("reload with invalid config: expected an error")
|
||||
}
|
||||
@@ -97,7 +99,7 @@ func TestReloadValidationFailureDoesNotApply(t *testing.T) {
|
||||
|
||||
// A subsequent valid reload clears the error.
|
||||
writeConfig(t, path, minimalConfigTOML("widget"))
|
||||
if err := controller.Reload(); err != nil {
|
||||
if err := controller.Reload(t.Context()); err != nil {
|
||||
t.Fatalf("recovery reload: %v", err)
|
||||
}
|
||||
if st := controller.Status(); st.LastError != "" || st.LastReloadAt == "" {
|
||||
@@ -105,6 +107,33 @@ func TestReloadValidationFailureDoesNotApply(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestReloadApplyFailureIsRecorded pins the other half of the safety
|
||||
// property: a file that loads cleanly can still fail to apply, because the
|
||||
// database layer reloadConfig re-applies comes from the control plane. That
|
||||
// failure must be recorded and returned, not reported as a successful
|
||||
// reload.
|
||||
func TestReloadApplyFailureIsRecorded(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "config.toml")
|
||||
writeConfig(t, path, minimalConfigTOML("widget"))
|
||||
|
||||
loader := configuration.New(slog.New(slog.DiscardHandler))
|
||||
controller := newReloadController(loader, path, "", func(context.Context, *configuration.Document) error {
|
||||
return errors.New("runtime settings unavailable")
|
||||
})
|
||||
|
||||
if err := controller.Reload(t.Context()); err == nil {
|
||||
t.Fatal("reload with a failing apply: expected an error")
|
||||
}
|
||||
st := controller.Status()
|
||||
if st.LastError == "" || st.LastErrorAt == "" {
|
||||
t.Fatalf("Status after a failed apply = %+v, want LastError and LastErrorAt set", st)
|
||||
}
|
||||
if st.LastReloadAt != "" {
|
||||
t.Fatalf("Status after a failed apply recorded a reload time: %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReloadLoopAppliesOnSignalAndStopsOnCancel(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "config.toml")
|
||||
@@ -112,8 +141,9 @@ func TestReloadLoopAppliesOnSignalAndStopsOnCancel(t *testing.T) {
|
||||
|
||||
loader := configuration.New(slog.New(slog.DiscardHandler))
|
||||
applied := make(chan struct{}, 1)
|
||||
controller := newReloadController(loader, path, "", func(*configuration.Document) {
|
||||
controller := newReloadController(loader, path, "", func(context.Context, *configuration.Document) error {
|
||||
applied <- struct{}{}
|
||||
return nil
|
||||
})
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
|
||||
Reference in new issue
Block a user