feat(container): apply container-runtime-policies live

The container runtime policies kind was restart-required because the pool
captured image, pull policy, network, max uptime and max concurrency at
construction and the dispatcher fixed its slot count at the start of a drain
pass. Both consumers now re-read the published config:

- the pool's settings come from a live source read per acquire, with the
  boot-resolved registry credential and Docker construction values preserved;
  a live default-image change is pulled rather than assumed present;
- the daemon owns one long-lived dispatcher that accepts a new limit: running
  tasks keep their slots, a raise admits queued work immediately, and a lower
  limit is applied as tasks finish;
- the control-plane descriptor is live, the kind joins runtimeResourceKinds,
  and the apply/reload paths resize the dispatcher after publishing. The
  ContainerConfig fields whose consumers now re-read are added to the reload
  allowlist; registry_auth stays restart-required.

Regenerate the docs artifacts. contracts.json also carries the voice
media-type description a prior commit left stale.
This commit is contained in:
sam committed 2026-10-02 11:39:46 +10:00
1 parent a8843c52b7
commit 04fa1b74fb
23 files changed
+573 -105

No files matched your search

@@ -170,14 +170,14 @@ looking at before reporting one:
| Key | Effective behaviour |
| ----------------- | ------------------------------------------------------------------------------------------------------------------ |
| `image` | Required only when enabled. |
| `max_concurrency` | Zero means no limit; used by general task dispatcher and container pool. |
| `max_uptime` | Default `60m` only when enabled; caps container lifetime. |
| `volume_ttl` | Default `72h` only when enabled; negative rejected. |
| `pull_policy` | Default `missing` when enabled. Pool code recognises `missing` and `always`; other strings silently skip pre-pull. |
| `registry_auth` | `SecretRef` (file-owned, `json:"-"`). Resolved at boot by `archied`'s `resolveRegistryAuth` and passed to `container.Config.RegistryAuth`; the pool base64url-encodes it into the `X-Registry-Auth` header (`internal/container/pool.go`'s `registryAuthHeader`). Zero ref = anonymous pull. Unresolvable/empty credential or an unencodable one warns and pulls anonymously rather than failing the pool. A half-named ref is refused by `configuration.validateContainers`. Preserved across the `container-runtime-policies` layering (the document cannot carry a secret ref), unlike `image`/`pull_policy`. |
| `network` | Empty uses best-effort self-network detection. Repository overlay pins `archie-core_default`. |
| `profiles` | Named agent execution environments (image or Kit); resolved per task dispatch (`ContainerConfig.Profile`). Own control-plane resource `agent-profiles` (`json:"-"` on the field so `container-runtime-policies` never carries it), `ApplyMode: live` -- a profile change applies without a restart, unlike the rest of `[containers]`. |
| `image` | Required at boot. Read by the container pool on every acquire (`containerLiveSettings` -> `Pool.current`, archie-core-zfb0.2), so a stored change applies without a restart. |
| `max_concurrency` | Zero means no limit. Read by the pool per acquire and applied to the daemon's running task dispatcher on publish (`ResizeTaskDispatcher`). |
| `max_uptime` | Default `60m` at boot; caps container lifetime. Read by the pool per acquire, so a stored change reaps the next container under the new cap. |
| `volume_ttl` | Default `72h` only when enabled; negative rejected. Re-read per cleanup cycle. |
| `pull_policy` | Default `missing` at boot. Pool code recognises `missing` and `always`; other strings silently skip pre-pull. Read by the pool per acquire. |
| `registry_auth` | `SecretRef` (file-owned, `json:"-"`). Resolved at boot by `archied`'s `resolveRegistryAuth` and passed to `container.Config.RegistryAuth`; the pool base64url-encodes it into the `X-Registry-Auth` header (`internal/container/pool.go`'s `registryAuthHeader`). Zero ref = anonymous pull. Unresolvable/empty credential or an unencodable one warns and pulls anonymously rather than failing the pool. A half-named ref is refused by `configuration.validateContainers`. `container-runtime-policies` cannot carry a secret ref, so this stays boot-resolved and requires-restart, unlike `image`/`pull_policy`. |
| `network` | Empty uses best-effort self-network detection at boot; the repository overlay pins `archie-core_default`. Read by the pool per acquire, falling back to the boot-resolved network when the live value is empty. |
| `profiles` | Named agent execution environments (image or Kit); resolved per task dispatch (`ContainerConfig.Profile`). Own control-plane resource `agent-profiles` (`json:"-"` on the field so `container-runtime-policies` never carries it), `ApplyMode: live`. |
| `credentials` | Maps a Kit `credential@1` service to a secret (`config.CredentialBinding`). Own control-plane resource `credential-bindings`, `ApplyMode: live`, `json:"-"` on the field. `kitrun.Launcher` reads it fresh from the live `config.Holder` on every `Launch`, never a value captured at boot -- see `internal/infrastructure/kitrun/launcher.go`'s `resolveCredentials`. A binding alone grants nothing; see `org`/`granted_credentials` below. |
| `org` / `granted_credentials` (root and per-identity) | An identity's org and the `credentials` service names it may resolve. `daemon.runKitTask` reads both fresh per dispatch (`d.configFor(task)`) and passes them to `kitrun.Request`; `config.ContainerConfig.BoundCredentials` computes the declared (Kit) ∩ granted (identity) ∩ org-matched intersection -- neither declared nor granted widens the other. |
+20 -13
View File
@@ -335,21 +335,28 @@ Two properties matter to anyone changing this area:
query, so boot records the last published settings and the reload re-applies
them from there.
- **Live kinds arrive on a watch.** Provider settings, model role assignments,
repository policies, scheduling policy, plugin settings, review settings and
channel settings are `ApplyMode: live`: the daemon keeps a watch per kind and,
on a new version, re-runs `boot.runtimeConfig` over the published snapshot and
repository policies, scheduling policy, plugin settings, review settings,
channel settings and container runtime policies are `ApplyMode: live`: the
daemon keeps a watch per kind (`boot.startRuntimeResourceWatches`,
`runtimeResourceKinds`) and, on a new version, re-runs `boot.runtimeConfig`
over the published snapshot and
republishes through `config.Holder`. A candidate the layered document cannot
run is refused before it is published — last-known-good keeps running and
the refusal is reported through apply status. A kind whose consumer lives in
another process reconciles there too: `archie-messaging` re-reads the stored
channel settings on the apply-status restamp interval and restarts only the
channel whose transport settings changed, while the listen addresses it has
bound and the set of channels it composed stay process bindings that take
effect on restart. A kind joins the live list only with a consumer that
re-reads it; one whose startup-built consumer never does stays
restart-required, and the model/provider half of the live set also rebuilds
the gateway chat runtime's provider set and the chat model list the chat
surfaces offer.
the refusal is reported through apply status. Container runtime policies
join the live set because their consumers re-read the published config: the
container pool reads image, pull policy, network, max uptime and concurrency
cap on every acquire and the daemon resizes its running task dispatcher from
the same publish; the fields the document cannot carry, `registry_auth`
above all, stay boot-resolved and remain requires-restart. A kind whose
consumer lives in another process reconciles there too: `archie-messaging`
re-reads the stored channel settings on the apply-status restamp interval
and restarts only the channel whose transport settings changed, while the
listen addresses it has bound and the set of channels it composed stay
process bindings that take effect on restart. A kind joins the live list
only with a consumer that re-reads it; one whose startup-built consumer
never does stays restart-required, and the model/provider half of the live
set also rebuilds the gateway chat runtime's provider set and the chat model
list the chat surfaces offer.
## Completion criteria
+1 -1
View File
@@ -350,7 +350,7 @@
"type": "object"
},
"type": {
"description": "Type is \"image\", \"video\", \"audio\", or \"document\".",
"description": "Type is \"image\", \"video\", \"audio\", \"voice\", or \"document\".",
"type": "string"
},
"url": {
+1 -1
View File
@@ -28,7 +28,7 @@
"url": "/dev/architecture/configuration/",
"title": "Configuration",
"section": "Architecture",
"body": "# Configuration\n\n**Status:** Approved; existing-type assignment is ongoing \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Ownership\n\nConfiguration has two separate responsibilities:\n\n1. `internal/infrastructure/configuration` loads files, environment variables,\n secret references, overlays, and compatibility representations.\n2. Each domain and plugin owns the typed runtime settings required for its\n behaviour.\n\n`internal/app` translates external input into those settings, validates\ncross-capability combinations, and supplies each component only its own\nsettings.\n\nThe following are prohibited:\n\n- passing complete application configuration into a domain;\n- making a domain depend on TOML, YAML, environment-variable names, secret\n references, or a configuration library;\n- defining a central runtime configuration type imported across the system;\n- placing plugin options in an unrelated global configuration model.\n\n## Private input document\n\n`internal/infrastructure/configuration` MAY use a complete decoded document\nwhile discovering files, decoding sections, and applying overlays. It is an\ninput-boundary DTO only.\n\nThe document:\n\n- remains private to the configuration infrastructure;\n- is composed from separately defined feature sections;\n- is immediately translated by application composition;\n- is never returned as the application's runtime model;\n- is never passed through runtime components.\n\nExternal DTOs and domain runtime settings are separate types.\n\nPlugins supply external configuration definitions through a narrow extension\ncontract. The core loader MUST NOT gain a field or recognised filename for every\nplugin.\n\nInvalid configuration input MUST NOT replace active valid settings or terminate\nunaffected capabilities. Configuration changes use the candidate validation,\npromotion, audit, and recovery requirements in\n[safe-change-and-recovery.md](/dev/architecture/safe-change-and-recovery/).\n\n## Dissolution of `internal/config`\n\n`internal/config` has no target location. It MUST be dissolved and MUST NOT\nsurvive as a shared package, configuration domain, or renamed global model.\n\nIts loading behaviour moves to `internal/infrastructure/configuration`. Existing\ntypes and methods are reassigned by behaviour. Repository execution policy,\ndispatch policy, retry policy, ecosystem defaults, tool policy, memory settings,\nprovider settings, and channel settings move to their\nrespective owners.\n\n## Repository settings\n\nThe existing `config.Repo` MUST NOT move intact into a repository domain or\nshared settings package. A managed repository is an application association:\n\n- work intake owns which repository references an identity monitors;\n- task execution owns gates, preflight, test protection, ecosystem behaviour,\n and change limits;\n- task lifecycle owns retry behaviour;\n- scheduling owns concurrency policy;\n- workspace management owns base branch and persistent-storage requirements.\n\nApplication composition correlates these settings by repository reference. No\ndomain receives a universal repository profile.\n\n## Documentation\n\nGenerated configuration reference describes external sections and their owning\ncapabilities. It MUST NOT establish a global Go configuration structure as the\npublic contract.\n\n## Value classification rule\n\nEvery value that controls behaviour MUST have an explicit owner and MUST be\nclassified as exactly one of:\n\n1. **Invariant** — changing it changes the meaning of the program, a wire\n protocol, or a domain concept. It may be a Go constant.\n2. **Runtime setting** — an operator, identity, domain, plugin, or deployment\n may legitimately choose a different value. It is typed configuration owned by\n the affected capability.\n3. **Policy value** — it expresses what is allowed, limited, routed, retried,\n protected, or gated. Its typed definition belongs to the owning domain and it\n is evaluated through the policy system.\n4. **Derived value** — it is computed from other settings and is not\n independently mutable.\n5. **Runtime state** — it changes as the application operates, such as the\n active model for a conversation. It is changed by a domain command and is not\n configuration merely because it can be changed.\n\nAn unclassified behavioural literal is a defect.\n\n`const` MUST NOT be used merely because a value currently has one default.\nConstants are reserved for actual invariants: enum discriminants, protocol\nversions, fixed compatibility tokens, and values whose alteration would change\nthe program's semantics rather than configure an installation.\n\nIn Go, a constant is a compile-time value and does not represent mutable runtime\nstorage. It generally has no address that can be observed or changed.\nArchitecturally, a constant therefore asserts that the value cannot change for\nthe lifetiLine truncated
"body": "# Configuration\n\n**Status:** Approved; existing-type assignment is ongoing \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Ownership\n\nConfiguration has two separate responsibilities:\n\n1. `internal/infrastructure/configuration` loads files, environment variables,\n secret references, overlays, and compatibility representations.\n2. Each domain and plugin owns the typed runtime settings required for its\n behaviour.\n\n`internal/app` translates external input into those settings, validates\ncross-capability combinations, and supplies each component only its own\nsettings.\n\nThe following are prohibited:\n\n- passing complete application configuration into a domain;\n- making a domain depend on TOML, YAML, environment-variable names, secret\n references, or a configuration library;\n- defining a central runtime configuration type imported across the system;\n- placing plugin options in an unrelated global configuration model.\n\n## Private input document\n\n`internal/infrastructure/configuration` MAY use a complete decoded document\nwhile discovering files, decoding sections, and applying overlays. It is an\ninput-boundary DTO only.\n\nThe document:\n\n- remains private to the configuration infrastructure;\n- is composed from separately defined feature sections;\n- is immediately translated by application composition;\n- is never returned as the application's runtime model;\n- is never passed through runtime components.\n\nExternal DTOs and domain runtime settings are separate types.\n\nPlugins supply external configuration definitions through a narrow extension\ncontract. The core loader MUST NOT gain a field or recognised filename for every\nplugin.\n\nInvalid configuration input MUST NOT replace active valid settings or terminate\nunaffected capabilities. Configuration changes use the candidate validation,\npromotion, audit, and recovery requirements in\n[safe-change-and-recovery.md](/dev/architecture/safe-change-and-recovery/).\n\n## Dissolution of `internal/config`\n\n`internal/config` has no target location. It MUST be dissolved and MUST NOT\nsurvive as a shared package, configuration domain, or renamed global model.\n\nIts loading behaviour moves to `internal/infrastructure/configuration`. Existing\ntypes and methods are reassigned by behaviour. Repository execution policy,\ndispatch policy, retry policy, ecosystem defaults, tool policy, memory settings,\nprovider settings, and channel settings move to their\nrespective owners.\n\n## Repository settings\n\nThe existing `config.Repo` MUST NOT move intact into a repository domain or\nshared settings package. A managed repository is an application association:\n\n- work intake owns which repository references an identity monitors;\n- task execution owns gates, preflight, test protection, ecosystem behaviour,\n and change limits;\n- task lifecycle owns retry behaviour;\n- scheduling owns concurrency policy;\n- workspace management owns base branch and persistent-storage requirements.\n\nApplication composition correlates these settings by repository reference. No\ndomain receives a universal repository profile.\n\n## Documentation\n\nGenerated configuration reference describes external sections and their owning\ncapabilities. It MUST NOT establish a global Go configuration structure as the\npublic contract.\n\n## Value classification rule\n\nEvery value that controls behaviour MUST have an explicit owner and MUST be\nclassified as exactly one of:\n\n1. **Invariant** — changing it changes the meaning of the program, a wire\n protocol, or a domain concept. It may be a Go constant.\n2. **Runtime setting** — an operator, identity, domain, plugin, or deployment\n may legitimately choose a different value. It is typed configuration owned by\n the affected capability.\n3. **Policy value** — it expresses what is allowed, limited, routed, retried,\n protected, or gated. Its typed definition belongs to the owning domain and it\n is evaluated through the policy system.\n4. **Derived value** — it is computed from other settings and is not\n independently mutable.\n5. **Runtime state** — it changes as the application operates, such as the\n active model for a conversation. It is changed by a domain command and is not\n configuration merely because it can be changed.\n\nAn unclassified behavioural literal is a defect.\n\n`const` MUST NOT be used merely because a value currently has one default.\nConstants are reserved for actual invariants: enum discriminants, protocol\nversions, fixed compatibility tokens, and values whose alteration would change\nthe program's semantics rather than configure an installation.\n\nIn Go, a constant is a compile-time value and does not represent mutable runtime\nstorage. It generally has no address that can be observed or changed.\nArchitecturally, a constant therefore asserts that the value cannot change for\nthe lifetiLine truncated
},
{
"url": "/dev/architecture/dashboard-agent-tools/",
+1 -1
View File
@@ -860,7 +860,7 @@ func (b *boot) startEmbeddedNATS(ctx context.Context) (string, string, error) {
// not require Docker; repository tasks park instead of falling back to a host
// model loop.
func (b *boot) setupContainers(ctx context.Context) func() {
containerPool, storeBackend, closeDocker := startContainers(ctx, b.cfg, b.secrets, b.log)
containerPool, storeBackend, closeDocker := startContainers(ctx, b.cfgHolder, b.secrets, b.log)
b.containerPool = containerPool
b.storeBackend = storeBackend
return closeDocker
+1 -1
View File
@@ -109,7 +109,7 @@ func TestConnectEmbeddedNATSUsesManagedWorkerBridge(t *testing.T) {
DockerClient: dockerClient,
Network: "workers",
RequireHostGateway: true,
}, slog.New(slog.DiscardHandler))
}, nil, slog.New(slog.DiscardHandler))
if err != nil {
t.Fatalf("container.NewPool = %v", err)
}
+27 -2
View File
@@ -92,6 +92,10 @@ func (b *boot) reloadConfig(ctx context.Context, doc *configuration.Document) er
old := b.cfgHolder.Get()
b.currentProvenance.Store(&doc.Provenance)
b.publishConfig(ctx, cfg)
// The container pool reads the published config on every acquire, but the
// dispatcher holds its limit; a reloaded containers.max_concurrency must
// reach it the same way a control-plane update does.
b.resizeTaskDispatcher(cfg.Containers.MaxConcurrency)
if fields := changedNonReloadableFields(old, cfg); len(fields) > 0 {
b.log.Warn("config reloaded; some changes require a restart",
"fields", fields, "paths", doc.Provenance.Paths())
@@ -260,10 +264,14 @@ func (b *boot) startLiveSettings(ctx context.Context) error {
// (b.runtimeConfig, which applies the execution budgets the settings watch
// published) and republishes through config.Holder, so the change takes
// effect without a restart. Kinds stay out of this list while a
// startup-built component still holds their value -- tool and container
// startup-built component still holds their value -- tool
// settings are frozen in components built at boot and remain
// restart-required -- and a kind joins it only with a consumer that re-reads
// it. PluginSettingsKind is on the list because its only consumer is the
// it. ContainerRuntimePoliciesKind is on the list because both consumers
// re-read the published config: the container pool takes its image, pull
// policy, network, max-uptime and concurrency cap per acquire, and the
// daemon's dispatcher is resized after the publish (applyRuntimeResourceUpdate
// below). PluginSettingsKind is on the list because its only consumer is the
// directory reconciliation, which reads the running config each tick; a
// removed plugin or engine still cannot unload, so that kind reports the
// removal rather than requiring a restart. ChannelSettingsKind is on the list
@@ -279,6 +287,7 @@ var runtimeResourceKinds = []string{
controlplane.PluginSettingsKind,
controlplane.ReviewSettingsKind,
controlplane.ChannelSettingsKind,
controlplane.ContainerRuntimePoliciesKind,
}
// startRuntimeResourceWatches keeps a watch per live kind established for the
@@ -362,6 +371,22 @@ func (b *boot) applyRuntimeResourceUpdate(ctx context.Context, kind string, upda
// report the reconciliation's own outcome, which is the one that knows
// whether a removal is outstanding.
b.reconcileRuntimePlugins(ctx)
case controlplane.ContainerRuntimePoliciesKind:
// The pool reads the republished config itself; the dispatcher holds
// its limit, so a raise must be woken to admit queued work now and a
// lower must be honoured as running tasks finish.
b.resizeTaskDispatcher(cfg.Containers.MaxConcurrency)
}
}
// resizeTaskDispatcher applies a new containers.max_concurrency to the
// daemon's running dispatcher. The pool needs no poke because it reads the
// published config on every acquire. b.d is nil in the boot window in which
// the watches are already live but the daemon is not built yet; there the
// dispatcher is created later from the config the next drain reads.
func (b *boot) resizeTaskDispatcher(maxConcurrency int) {
if b.d != nil {
b.d.ResizeTaskDispatcher(maxConcurrency)
}
}
@@ -15,6 +15,7 @@ import (
"github.com/samcharles93/archie-core/internal/agentexec"
"github.com/samcharles93/archie-core/internal/app/controlplane"
pb "github.com/samcharles93/archie-core/internal/contracts/controlplane/v1"
"github.com/samcharles93/archie-core/internal/daemon"
"github.com/samcharles93/archie-core/internal/secret"
)
@@ -240,3 +241,33 @@ func TestLiveModelRoleUpdateRebuildsTheChatModelRuntime(t *testing.T) {
})
}
}
// TestLiveContainerRuntimePoliciesReLayerAndResizeTheDispatcher pins the
// container kind's live seam (archie-core-zfb0.2) at the apply path: a stored
// container-runtime-policies document re-layers into the running config the
// pool reads per acquire, and the daemon resizes its running dispatcher from
// the same publish rather than requiring a restart.
func TestLiveContainerRuntimePoliciesReLayerAndResizeTheDispatcher(t *testing.T) {
values := databaseOwnedResources()
values[controlplane.ContainerRuntimePoliciesKind] = map[string]any{
"image": "archie:live", "max_concurrency": 5, "pull_policy": "missing",
}
stub := &resourceWatchStub{values: values, version: 3, storedVersion: map[string]int64{}, opens: map[string]int{}}
b, _ := newResourceWatchBoot(t, stub)
if err := b.loadRuntimeConfig(t.Context()); err != nil {
t.Fatalf("loadRuntimeConfig: %v", err)
}
// A running daemon, so the resize arm executes against a real dispatcher
// rather than taking the boot-window nil guard.
b.d = &daemon.Daemon{Cfg: b.cfgHolder, Log: b.log}
b.applyRuntimeResourceUpdate(t.Context(), controlplane.ContainerRuntimePoliciesKind, controlplane.AppliedResource{Version: 4})
got := b.cfgHolder.Get().Containers
if got.MaxConcurrency != 5 {
t.Errorf("running max_concurrency = %d, want the stored 5", got.MaxConcurrency)
}
if got.Image != "archie:live" {
t.Errorf("running image = %q, want the stored archie:live", got.Image)
}
}
+23 -2
View File
@@ -974,11 +974,12 @@ func safePluginInfo(p plugin.Plugin) (name, version string) {
// there is deliberately no host execution fallback.
func startContainers(
ctx context.Context,
cfg config.Config,
cfgHolder *config.Holder,
secrets *secret.Registry,
log *slog.Logger,
) (*container.Pool, storage.Backend, func()) {
noop := func() {}
cfg := cfgHolder.Get()
// Single Docker client shared between pool and storage backend.
dockerCli, err := client.New(client.FromEnv)
if err != nil {
@@ -1002,7 +1003,7 @@ func startContainers(
// Only the embedded broker binds a discovered host gateway. The
// standalone State Store uses its configured, container-reachable target.
RequireHostGateway: cfg.NATS.Mode == config.NATSModeEmbedded,
}, log)
}, containerLiveSettings(cfgHolder), log)
if err != nil {
// A missing image is recoverable by hand.
log.Error("autonomous workflows unavailable: container pool unavailable", "err", err,
@@ -1021,6 +1022,26 @@ func startContainers(
}
}
// containerLiveSettings reads the per-acquire container settings from the
// running config. The pool calls it on every acquire, so a stored
// container-runtime-policies change reaches the next container without a
// restart. Only fields the resource document carries are taken live:
// RegistryAuth is file-owned and stays the boot-resolved value, DockerClient
// and RequireHostGateway are construction-time, and GracePeriod is not part
// of the document.
func containerLiveSettings(cfgHolder *config.Holder) func() container.Config {
return func() container.Config {
cfg := cfgHolder.Get()
return container.Config{
Image: cfg.Containers.Image,
MaxConcurrency: cfg.Containers.MaxConcurrency,
MaxUptime: cfg.Containers.MaxUptime.Std(),
PullPolicy: cfg.Containers.PullPolicy,
Network: cfg.Containers.Network,
}
}
}
// resolveRegistryAuth resolves [containers].registry_auth into the credential
// the container pool sends on pull. It returns "" -- an anonymous pull, the
// pool's behaviour before the field existed -- whenever the credential is not
+12 -8
View File
@@ -156,17 +156,21 @@ var reloadableFields = map[string]bool{
// "Selection": a Kit profile applies without a restart. Credentials is the
// same shape one layer down: kitrun.Launcher reads Config.Get().Containers.
// Credentials fresh on every Launch (CredentialBindingsKind), never a slice
// captured once when the launcher was built. Everything else in
// Containers (Image, MaxConcurrency, MaxUptime, PullPolicy, Network,
// RegistryAuth) is frozen in the startup-built container pool -- the
// dispatchers re-read MaxConcurrency but the pool captures it at construction
// (container/pool.go:94,163), so a change only partially applies and must
// warn requires-restart. Forge.Host is carried into TaskContext by ForTask
// captured once when the launcher was built. Image, MaxConcurrency, MaxUptime,
// PullPolicy and Network are re-read per acquire by the container pool and
// MaxConcurrency also resizes the running dispatcher on publish
// (archie-core-zfb0.2), so a reload applies them without a restart.
// RegistryAuth stays frozen in the startup-built pool: it is file-owned and
// never part of the container-runtime-policies document, so a changed value
// must warn requires-restart. Forge.Host is carried into TaskContext by ForTask
// (display/link building only); the forge client itself is startup-built, so
// Type/Token/TokenEnv stay requires-restart.
var reloadableSubFields = map[string]map[string]bool{
"Containers": {"VolumeTTL": true, "Profiles": true, "Credentials": true},
"Forge": {"Host": true},
"Containers": {
"VolumeTTL": true, "Profiles": true, "Credentials": true,
"Image": true, "MaxConcurrency": true, "MaxUptime": true, "PullPolicy": true, "Network": true,
},
"Forge": {"Host": true},
// Policy is carried into TaskConfig by ForTask (config.go); MCPServers,
// WebFetch and Minimax are not and stay requires-restart.
"Tools": {"Policy": true},
+12 -9
View File
@@ -198,28 +198,31 @@ func TestChangedNonReloadableFields(t *testing.T) {
t.Fatalf("reloadable-only: got %v, want []", got)
}
// Containers sub-field granularity: Image change warns, MaxConcurrency
// and VolumeTTL changes do not.
// Containers sub-field granularity: the fields the pool re-reads per
// acquire, and the concurrency cap the dispatcher resizes on, are
// reloadable; RegistryAuth stays frozen in the startup-built pool.
c1 := base
c1.Containers = config.ContainerConfig{Image: "img1", MaxConcurrency: 2, VolumeTTL: config.Duration(24 * time.Hour)}
c2 := c1
c2.Containers.Image = "img2"
if got := changedNonReloadableFields(c1, c2); len(got) != 1 || got[0] != "Containers.Image" {
t.Fatalf("Image change: got %v, want [Containers.Image]", got)
if got := changedNonReloadableFields(c1, c2); len(got) != 0 {
t.Fatalf("Image change: got %v, want [] (the pool reads it per acquire)", got)
}
c3 := c1
c3.Containers.MaxConcurrency = 4
// MaxConcurrency is NOT reloadable: the dispatchers re-read it but the
// startup-built container pool captures it (container/pool.go:94), so
// a change only partially applies and must warn requires-restart.
if got := changedNonReloadableFields(c1, c3); len(got) != 1 || got[0] != "Containers.MaxConcurrency" {
t.Fatalf("MaxConcurrency change: got %v, want [Containers.MaxConcurrency]", got)
if got := changedNonReloadableFields(c1, c3); len(got) != 0 {
t.Fatalf("MaxConcurrency change: got %v, want [] (the pool reads it and the dispatcher resizes)", got)
}
c4 := c1
c4.Containers.VolumeTTL = config.Duration(48 * time.Hour)
if got := changedNonReloadableFields(c1, c4); len(got) != 0 {
t.Fatalf("VolumeTTL change: got %v, want []", got)
}
c5 := c1
c5.Containers.RegistryAuth = config.SecretRef{Engine: "env", Key: "REGISTRY_AUTH"}
if got := changedNonReloadableFields(c1, c5); len(got) != 1 || got[0] != "Containers.RegistryAuth" {
t.Fatalf("RegistryAuth change: got %v, want [Containers.RegistryAuth] (the pool resolves it at boot)", got)
}
// NATS is not reloadable: a URL change warns (the daemon's client is
// startup-built; containers get the frozen ConnectedNATS).
@@ -47,7 +47,7 @@ func TestExternalServicesDoNotRequireBridgeNetwork(t *testing.T) {
cfg.NATS.Mode = config.NATSModeExternal
cfg.Containers.Network = "host"
cfg.Services = config.Services{config.ServiceNameState: {Target: "state.example.test:9090"}}
pool, _, cleanup := startContainers(t.Context(), cfg, secret.NewRegistry(), slog.New(slog.DiscardHandler))
pool, _, cleanup := startContainers(t.Context(), config.NewHolder(cfg), secret.NewRegistry(), slog.New(slog.DiscardHandler))
defer cleanup()
if pool == nil {
t.Fatal("external services on host networking must construct a worker pool")
@@ -197,11 +197,11 @@ func TestRegistryRoutesAndValidatesDefinitions(t *testing.T) {
found := false
for _, descriptor := range catalog.Resources {
if descriptor.Kind == ContainerRuntimePoliciesKind {
found = descriptor.ApplyMode == "restart-required"
found = descriptor.ApplyMode == "live"
}
}
if !found {
t.Fatal("container runtime policies missing restart-required descriptor")
t.Fatal("container runtime policies missing live descriptor")
}
_, err = server.Command(t.Context(), &pb.CommandRequest{Kind: ContainerRuntimePoliciesKind, Command: "replace", ValueJson: []byte(`{"pull_policy":"sometimes"}`), Actor: "test", Source: "test", RequestId: "bad", ExpectedVersion: 1})
@@ -54,7 +54,13 @@ func operationalDefinitions() []Definition {
{Kind: PluginSettingsKind, Title: "Plugin settings", ApplyMode: "live", Document: pluginSettings{}, Seed: func(cfg config.Config) any {
return pluginSettings{cfg.PluginDir, cfg.ModuleDir, cfg.SecretEngineDir, cfg.SkillsDir}
}, Validate: validatePluginSettings},
{Kind: ContainerRuntimePoliciesKind, Title: "Container runtime policies", ApplyMode: "restart-required", Document: containerRuntimePolicies{}, Seed: seedContainerPolicies, Validate: validateContainers, Normalize: normalizeContainerPolicies},
// ApplyMode is live: the container pool reads the published config on
// every acquire (image, pull policy, network, max uptime, concurrency
// cap) and the daemon resizes its running dispatcher after the publish,
// so neither a deferred container nor a queued task needs a restart
// (archie-core-zfb0.2). RegistryAuth is file-owned and never part of
// this document, so it stays a boot-resolved value.
{Kind: ContainerRuntimePoliciesKind, Title: "Container runtime policies", ApplyMode: "live", Document: containerRuntimePolicies{}, Seed: seedContainerPolicies, Validate: validateContainers, Normalize: normalizeContainerPolicies},
}
}
@@ -553,13 +553,13 @@ func TestRuntimeResourceKindsApplyLive(t *testing.T) {
modes[definition.Kind] = definition.ApplyMode
}
for _, kind := range []string{
ProviderSettingsKind, ModelRoleAssignmentsKind, RepositoryPoliciesKind, SchedulingPolicyKind, AgentProfileKind, CredentialBindingsKind, PluginSettingsKind, ReviewSettingsKind, ChannelSettingsKind,
ProviderSettingsKind, ModelRoleAssignmentsKind, RepositoryPoliciesKind, SchedulingPolicyKind, AgentProfileKind, CredentialBindingsKind, PluginSettingsKind, ReviewSettingsKind, ChannelSettingsKind, ContainerRuntimePoliciesKind,
} {
if modes[kind] != "live" {
t.Errorf("%s applies %q, want live: the daemon re-layers this kind on a watch", kind, modes[kind])
}
}
for _, kind := range []string{ToolSettingsKind, ContainerRuntimePoliciesKind} {
for _, kind := range []string{ToolSettingsKind} {
if modes[kind] != "restart-required" {
t.Errorf("%s applies %q, want restart-required: a startup-built component still holds it", kind, modes[kind])
}
+2 -2
View File
@@ -56,7 +56,7 @@ func TestNewPoolResolvesDefaultBridgeGatewayForNativeDaemon(t *testing.T) {
pool, err := NewPool(t.Context(), Config{
DockerClient: dockerClient,
RequireHostGateway: true,
}, discardLogger())
}, nil, discardLogger())
if err != nil {
t.Fatalf("NewPool = %v", err)
}
@@ -138,7 +138,7 @@ func TestNewPoolRejectsUnreachableEmbeddedNetwork(t *testing.T) {
DockerClient: dockerClient,
Network: "workers",
RequireHostGateway: true,
}, discardLogger())
}, nil, discardLogger())
if err == nil {
_ = pool.Close()
t.Fatal("NewPool = nil error, want embedded reachability validation failure")
+94 -33
View File
@@ -98,6 +98,12 @@ type Pool struct {
// the listener to an address visible only to workers on this bridge.
hostGateway string
// live, when set, returns the pool's current settings. A production pool
// is given one that reads the running config, so a container-runtime-
// policies change reaches the next acquire without a restart; a test pool
// leaves it nil and cfg is the whole truth.
live func() Config
mu sync.Mutex
active int
// pulled records the images already made available, so a profile's image
@@ -108,6 +114,25 @@ type Pool struct {
teardowns map[string]*containerTeardown
}
// current returns the settings a new acquire should use: the live source when
// the pool has one, otherwise the construction-time Config. Only the fields a
// container-runtime-policies document can carry are taken from the live
// source; DockerClient, RequireHostGateway, RegistryAuth and GracePeriod stay
// construction-time, because none of them is part of that document.
func (p *Pool) current() Config {
if p.live == nil {
return p.cfg
}
cfg := p.cfg
live := p.live()
cfg.Image = live.Image
cfg.MaxConcurrency = live.MaxConcurrency
cfg.MaxUptime = live.MaxUptime
cfg.PullPolicy = live.PullPolicy
cfg.Network = live.Network
return cfg
}
// Config is the subset of daemon container configuration the pool needs.
type Config struct {
Image string
@@ -142,7 +167,14 @@ type Config struct {
//
// If cfg.DockerClient is set, it is reused (caller owns Close).
// Otherwise a new client is created via client.FromEnv (pool owns Close).
func NewPool(ctx context.Context, cfg Config, log *slog.Logger) (*Pool, error) {
//
// live, when non-nil, is the pool's live source of the per-acquire settings a
// container-runtime-policies document can change (Image, MaxConcurrency,
// MaxUptime, PullPolicy, Network). It is read on every acquire, so a stored
// change reaches the next container without a restart; nil means cfg is fixed
// for the pool's life. DockerClient, RequireHostGateway, RegistryAuth and
// GracePeriod stay construction-time because the document never carries them.
func NewPool(ctx context.Context, cfg Config, live func() Config, log *slog.Logger) (*Pool, error) {
cli := cfg.DockerClient
if cli == nil {
var err error
@@ -173,13 +205,16 @@ func NewPool(ctx context.Context, cfg Config, log *slog.Logger) (*Pool, error) {
p := &Pool{
cli: cli,
cfg: cfg,
live: live,
log: log,
ownCli: cfg.DockerClient == nil,
network: network,
hostGateway: hostGateway,
}
// Pull image if needed.
// Pull image if needed. The configured image is recorded as pulled, so an
// acquire that names it skips a second pull even though the live settings
// may have changed the configured image since.
if cfg.PullPolicy == "always" || cfg.PullPolicy == "missing" {
if err := p.pullImage(ctx, cfg.Image); err != nil {
if err := cli.Close(); err != nil {
@@ -187,6 +222,7 @@ func NewPool(ctx context.Context, cfg Config, log *slog.Logger) (*Pool, error) {
}
return nil, err
}
p.markPulled(cfg.Image)
}
// Recover orphaned containers from a previous daemon crash.
@@ -201,13 +237,17 @@ func NewPool(ctx context.Context, cfg Config, log *slog.Logger) (*Pool, error) {
// is set, the pool schedules a hard stop and remove of the container once
// that lifetime cap elapses, regardless of task state.
func (p *Pool) Acquire(ctx context.Context, image string, mounts []storage.Mount, env []string) (*Container, error) {
// One snapshot for the whole acquire: the image, pull policy, network,
// concurrency cap and max-uptime cap are all read once, so a settings
// change landing mid-acquire cannot mix two versions.
cfg := p.current()
if image == "" {
image = p.cfg.Image
image = cfg.Image
}
if err := p.EnsureImage(ctx, image); err != nil {
if err := p.ensureImage(ctx, cfg, image); err != nil {
return nil, err
}
if err := p.reserve(); err != nil {
if err := p.reserve(cfg.MaxConcurrency); err != nil {
return nil, err
}
@@ -217,12 +257,18 @@ func (p *Pool) Acquire(ctx context.Context, image string, mounts []storage.Mount
Mounts: storage.ConvertMounts(mounts),
AutoRemove: true,
}
if p.network != "" {
// The live network wins when it names one; otherwise the network resolved
// at construction (the configured one, or self-detection) stays in force.
network := cfg.Network
if network == "" {
network = p.network
}
if network != "" {
// Join the same user-defined network the daemon itself is on, so
// the agent container can resolve sibling compose services (nats,
// etc.) by name -- otherwise Docker attaches it to the default
// bridge network, where those hostnames don't resolve.
hostConfig.NetworkMode = container.NetworkMode(p.network)
hostConfig.NetworkMode = container.NetworkMode(network)
}
resp, err := p.cli.ContainerCreate(ctx, client.ContainerCreateOptions{
@@ -249,16 +295,17 @@ func (p *Pool) Acquire(ctx context.Context, image string, mounts []storage.Mount
p.unreserve()
return nil, fmt.Errorf("container start: %w", err)
}
return p.started(ctx, resp.ID, name), nil
return p.started(ctx, cfg.MaxUptime, resp.ID, name), nil
}
// AcquireKit starts a Kit task container under the same concurrency cap,
// lifetime cap and exit watch as Acquire. Release tears it down.
func (p *Pool) AcquireKit(ctx context.Context, s KitSpec) (*Container, error) {
if err := p.EnsureImage(ctx, s.Image); err != nil {
cfg := p.current()
if err := p.ensureImage(ctx, cfg, s.Image); err != nil {
return nil, err
}
if err := p.reserve(); err != nil {
if err := p.reserve(cfg.MaxConcurrency); err != nil {
return nil, err
}
s.Labels = maps.Clone(s.Labels)
@@ -271,14 +318,14 @@ func (p *Pool) AcquireKit(ctx context.Context, s KitSpec) (*Container, error) {
p.unreserve()
return nil, err
}
return p.started(ctx, id, s.Name), nil
return p.started(ctx, cfg.MaxUptime, id, s.Name), nil
}
func (p *Pool) reserve() error {
func (p *Pool) reserve(maxConcurrency int) error {
p.mu.Lock()
defer p.mu.Unlock()
if p.cfg.MaxConcurrency > 0 && p.active >= p.cfg.MaxConcurrency {
return fmt.Errorf("max concurrency %d reached", p.cfg.MaxConcurrency)
if maxConcurrency > 0 && p.active >= maxConcurrency {
return fmt.Errorf("max concurrency %d reached", maxConcurrency)
}
p.active++
return nil
@@ -290,9 +337,9 @@ func (p *Pool) unreserve() {
p.mu.Unlock()
}
func (p *Pool) started(ctx context.Context, id, name string) *Container {
if p.cfg.MaxUptime > 0 {
p.armMaxUptime(ctx, id)
func (p *Pool) started(ctx context.Context, maxUptime time.Duration, id, name string) *Container {
if maxUptime > 0 {
p.armMaxUptime(ctx, maxUptime, id)
}
p.log.Info("container started", "id", id[:12], "name", name)
return &Container{ID: id, exited: p.watchExit(ctx, id)}
@@ -348,12 +395,12 @@ type containerTeardown struct {
// The lock covers only bookkeeping: the callback runs on its own goroutine and
// nothing here waits on it, and every Docker call it makes happens after it
// has released p.mu.
func (p *Pool) armMaxUptime(ctx context.Context, id string) {
func (p *Pool) armMaxUptime(ctx context.Context, maxUptime time.Duration, id string) {
p.mu.Lock()
defer p.mu.Unlock()
t := p.teardownLocked(id)
t.timer = time.AfterFunc(p.cfg.MaxUptime, func() {
t.timer = time.AfterFunc(maxUptime, func() {
p.reapMaxUptime(ctx, id)
})
}
@@ -544,11 +591,18 @@ func (p *Pool) Close() error {
// ── helpers ──────────────────────────────────────────────────────────
// EnsureImage pulls an image other than the configured one the first time a
// task asks for it, under the configured pull policy. The configured image
// was already pulled by NewPool.
// EnsureImage pulls ref under the live pull policy unless the pool has
// already made it available. The configured image was pulled by NewPool and
// recorded, so a caller naming it skips a second pull.
func (p *Pool) EnsureImage(ctx context.Context, ref string) error {
if ref == p.cfg.Image || (p.cfg.PullPolicy != "always" && p.cfg.PullPolicy != "missing") {
return p.ensureImage(ctx, p.current(), ref)
}
// ensureImage pulls ref under cfg's pull policy unless it is already present
// in the pool. It is the live read an acquire uses; EnsureImage is the
// exported form for callers that only name a ref.
func (p *Pool) ensureImage(ctx context.Context, cfg Config, ref string) error {
if cfg.PullPolicy != "always" && cfg.PullPolicy != "missing" {
return nil
}
p.mu.Lock()
@@ -560,19 +614,26 @@ func (p *Pool) EnsureImage(ctx context.Context, ref string) error {
if err := p.pullImage(ctx, ref); err != nil {
return err
}
p.markPulled(ref)
return nil
}
// markPulled records that ref is available locally, so a later acquire skips
// a second pull.
func (p *Pool) markPulled(ref string) {
p.mu.Lock()
if p.pulled == nil {
p.pulled = map[string]bool{}
}
p.pulled[ref] = true
p.mu.Unlock()
return nil
}
// pullImage pulls ref. On "missing" policy, skips if the image already
// exists locally.
// pullImage pulls ref under the live pull policy. On "missing" policy, skips
// if the image already exists locally.
func (p *Pool) pullImage(ctx context.Context, ref string) error {
if p.cfg.PullPolicy == "missing" {
cfg := p.current()
if cfg.PullPolicy == "missing" {
_, err := p.cli.ImageInspect(ctx, ref)
if err == nil {
p.log.Info("image already present", "image", ref)
@@ -581,7 +642,7 @@ func (p *Pool) pullImage(ctx context.Context, ref string) error {
}
p.log.Info("pulling image", "image", ref)
rc, err := p.cli.ImagePull(ctx, ref, client.ImagePullOptions{RegistryAuth: p.registryAuthHeader(ref)})
rc, err := p.cli.ImagePull(ctx, ref, client.ImagePullOptions{RegistryAuth: p.registryAuthHeader(cfg, ref)})
if err != nil {
return fmt.Errorf("pull %s: %w", ref, err)
}
@@ -606,11 +667,11 @@ func (p *Pool) pullImage(ctx context.Context, ref string) error {
// autonomous task down with it, and the warning is what makes the 401 a private
// registry will still answer explicable. A missing credential is silent -- it is
// the deployment shape that has no private registry.
func (p *Pool) registryAuthHeader(ref string) string {
if p.cfg.RegistryAuth == "" {
func (p *Pool) registryAuthHeader(cfg Config, ref string) string {
if cfg.RegistryAuth == "" {
return ""
}
encoded, err := encodeRegistryAuth(p.cfg.RegistryAuth)
encoded, err := encodeRegistryAuth(cfg.RegistryAuth)
if err != nil {
p.log.Warn("registry credential unusable; pulling without auth", "image", ref, "err", err)
return ""
@@ -700,8 +761,8 @@ func (p *Pool) Active() int {
return p.active
}
// Cap is the concurrency cap this pool enforces, from the configured
// Cap is the concurrency cap this pool enforces, from the live
// containers.max_concurrency. Zero means unlimited (Acquire only enforces a
// cap when MaxConcurrency > 0), so a caller rendering "active/cap" must not
// treat zero as a cap of zero.
func (p *Pool) Cap() int { return p.cfg.MaxConcurrency }
func (p *Pool) Cap() int { return p.current().MaxConcurrency }
+156
View File
@@ -12,6 +12,7 @@ import (
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
@@ -844,3 +845,158 @@ func TestPullImageWithoutUsableRegistryCredentialPullsAnonymously(t *testing.T)
})
}
}
// ── live settings (archie-core-zfb0.2) ────────────────────────────────
// TestPoolReadsSettingsPerAcquire pins that a container-runtime-policies
// change reaches the next acquire without a restart. The pool's image, pull
// policy, network, concurrency cap and max-uptime cap must come from the live
// settings source rather than the value captured when the pool was built; a
// construction-time network stays the fallback when the live value is empty.
func TestPoolReadsSettingsPerAcquire(t *testing.T) {
type createBody struct {
Image string `json:"Image"`
HostConfig struct {
NetworkMode string `json:"NetworkMode"`
} `json:"HostConfig"`
}
var (
mu sync.Mutex
creates []createBody
pulls []string
)
var stopped, removed atomic.Int32
dockerAPI := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasSuffix(r.URL.Path, "/containers/create"):
var body createBody
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
mu.Lock()
creates = append(creates, body)
id := fmt.Sprintf("%064d", len(creates))
mu.Unlock()
writeDockerJSON(t, w, map[string]any{"Id": id, "Warnings": []string{}})
case strings.HasSuffix(r.URL.Path, "/start"):
w.WriteHeader(http.StatusNoContent)
case strings.HasSuffix(r.URL.Path, "/stop"):
stopped.Add(1)
w.WriteHeader(http.StatusNoContent)
case strings.HasSuffix(r.URL.Path, "/images/create"):
mu.Lock()
pulls = append(pulls, r.URL.Query().Get("fromImage"))
mu.Unlock()
writeDockerJSON(t, w, map[string]any{})
case strings.HasSuffix(r.URL.Path, "/json"):
http.Error(w, "no such image", http.StatusNotFound)
case r.Method == http.MethodDelete:
removed.Add(1)
w.WriteHeader(http.StatusNoContent)
default:
http.Error(w, "unexpected Docker API path "+r.URL.Path, http.StatusNotFound)
}
}))
t.Cleanup(dockerAPI.Close)
dockerClient, err := client.New(client.WithHost(dockerAPI.URL), client.WithAPIVersion("1.55"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = dockerClient.Close() })
var settingsMu sync.Mutex
current := Config{
Image: "old/image", PullPolicy: "missing", Network: "",
MaxConcurrency: 1, MaxUptime: 10 * time.Second,
}
setCurrent := func(c Config) {
settingsMu.Lock()
current = c
settingsMu.Unlock()
}
pool := &Pool{
cli: dockerClient,
cfg: current,
network: "detected",
live: func() Config {
settingsMu.Lock()
defer settingsMu.Unlock()
return current
},
// NewPool pulls the configured image before the first acquire.
pulled: map[string]bool{"old/image": true},
log: discardLogger(),
}
acquire := func(t *testing.T) {
t.Helper()
if _, err := pool.Acquire(context.Background(), "", nil, nil); err != nil {
t.Fatalf("Acquire: %v", err)
}
}
created := func(t *testing.T, i int) createBody {
t.Helper()
mu.Lock()
defer mu.Unlock()
if i >= len(creates) {
t.Fatalf("only %d create calls, want at least %d", len(creates), i+1)
}
return creates[i]
}
acquire(t)
if got := created(t, 0); got.Image != "old/image" || got.HostConfig.NetworkMode != "detected" {
t.Fatalf("first acquire used image %q network %q, want the construction settings", got.Image, got.HostConfig.NetworkMode)
}
setCurrent(Config{
Image: "new/image", PullPolicy: "missing", Network: "newnet",
MaxConcurrency: 2, MaxUptime: 60 * time.Millisecond,
})
acquire(t)
second := created(t, 1)
if second.Image != "new/image" {
t.Errorf("second acquire used image %q, want the live image", second.Image)
}
if second.HostConfig.NetworkMode != "newnet" {
t.Errorf("second acquire joined network %q, want the live network", second.HostConfig.NetworkMode)
}
if got := pool.Cap(); got != 2 {
t.Errorf("Cap() = %d, want the live MaxConcurrency 2", got)
}
mu.Lock()
pulledNew := false
for _, ref := range pulls {
if strings.HasSuffix(ref, "new/image") {
pulledNew = true
}
}
mu.Unlock()
if !pulledNew {
t.Errorf("pulls = %v, want the live image pulled", pulls)
}
// The live max-uptime cap, not the construction-time 10s, reaps it.
deadline := time.Now().Add(2 * time.Second)
for (stopped.Load() == 0 || removed.Load() == 0) && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
if stopped.Load() == 0 || removed.Load() == 0 {
t.Errorf("live max uptime did not reap the second container: stopped=%d removed=%d", stopped.Load(), removed.Load())
}
// An empty live network keeps the construction-time fallback in force.
setCurrent(Config{
Image: "new/image", PullPolicy: "missing", Network: "",
MaxConcurrency: 3, MaxUptime: 10 * time.Second,
})
acquire(t)
if got := created(t, 2); got.HostConfig.NetworkMode != "detected" {
t.Errorf("third acquire joined network %q, want the construction-time fallback %q", got.HostConfig.NetworkMode, "detected")
}
}
+1 -1
View File
@@ -68,7 +68,7 @@ func TestConnectedNATSEndpointReachesManagedContainerCreate(t *testing.T) {
Network: networkName,
DockerClient: dockerClient,
RequireHostGateway: true,
}, slog.New(slog.DiscardHandler))
}, nil, slog.New(slog.DiscardHandler))
if err != nil {
t.Fatalf("container.NewPool = %v", err)
}
+66 -15
View File
@@ -199,6 +199,12 @@ type Daemon struct {
// execution is unavailable and process parks tasks; every runnable task gets
// a fresh container.
ContainerPool *container.Pool
// dispatcher bounds task execution globally across poll passes. Built once,
// on first use, so ResizeTaskDispatcher can apply a live
// containers.max_concurrency change without a restart; drainNATS submits
// each pass through it and waits for that pass.
dispatcherOnce sync.Once
dispatcher *taskDispatcher
// Storage is the pluggable storage backend for container mounts.
// A runnable task requires it; acquireTaskContainer parks on nil.
Storage storage.Backend
@@ -975,7 +981,7 @@ func (d *Daemon) botUserForTask(task *workflowtask.Task) string {
// requeued tasks (waiting_human approval, retry-parked) that didn't come
// through a NATS publish.
func (d *Daemon) drainNATS(ctx context.Context) {
dispatcher := newTaskDispatcher(d.Cfg.Get().Containers.MaxConcurrency, d.allowConcurrentForTask)
dispatcher := d.taskDispatcher()
for ctx.Err() == nil {
msg, err := d.Tasks.Fetch(ctx)
if err != nil && !errors.Is(err, eventbus.ErrNoMessage) {
@@ -1004,6 +1010,25 @@ func (d *Daemon) drainNATS(ctx context.Context) {
dispatcher.Wait()
}
// taskDispatcher returns the daemon's long-lived global dispatcher, building
// it from the running config on first use. It outlives a single drain pass so
// ResizeTaskDispatcher can resize the running dispatcher; drainNATS still
// waits for each pass's work before returning.
func (d *Daemon) taskDispatcher() *taskDispatcher {
d.dispatcherOnce.Do(func() {
d.dispatcher = newTaskDispatcher(d.Cfg.Get().Containers.MaxConcurrency, d.allowConcurrentForTask)
})
return d.dispatcher
}
// ResizeTaskDispatcher applies a new containers.max_concurrency to the
// running global task dispatcher without a restart. Running tasks keep their
// slots; a lowered limit is honoured as they finish, and a raised limit
// admits queued work immediately.
func (d *Daemon) ResizeTaskDispatcher(maxConcurrency int) {
d.taskDispatcher().SetMaxConcurrency(maxConcurrency)
}
// submitNATSTask decodes a fetched message and queues it for processing. A
// message that cannot be decoded is acked rather than redelivered: it will
// never decode on a retry, so leaving it unacked would block the queue.
@@ -1026,27 +1051,55 @@ func (d *Daemon) submitNATSTask(ctx context.Context, dispatcher *taskDispatcher,
// allowConcurrent reports true opt out of the same-repo serialization
// entirely -- the global slot limit is the only bound on their concurrency.
type taskDispatcher struct {
slots chan struct{}
allowConcurrent func(task *workflow.Task) bool
mu sync.Mutex
repoTail map[string]chan struct{}
wg sync.WaitGroup
mu sync.Mutex
limit int
active int
slotsFree *sync.Cond
repoTail map[string]chan struct{}
wg sync.WaitGroup
}
// SetMaxConcurrency accepts a new global limit without a restart. Running
// tasks keep their slots; a lowered limit is enforced as they finish, and a
// raised limit admits queued work immediately.
func (d *taskDispatcher) SetMaxConcurrency(maxConcurrency int) {
d.mu.Lock()
d.limit = maxConcurrency
d.mu.Unlock()
d.slotsFree.Broadcast()
}
func newTaskDispatcher(maxConcurrency int, allowConcurrent func(task *workflow.Task) bool) *taskDispatcher {
var slots chan struct{}
if maxConcurrency > 0 {
slots = make(chan struct{}, maxConcurrency)
}
if allowConcurrent == nil {
allowConcurrent = func(*workflow.Task) bool { return false }
}
return &taskDispatcher{
slots: slots,
d := &taskDispatcher{
limit: maxConcurrency,
allowConcurrent: allowConcurrent,
repoTail: make(map[string]chan struct{}),
}
d.slotsFree = sync.NewCond(&d.mu)
return d
}
// acquireSlot blocks until the global limit has room and returns the release
// function. limit <= 0 means unlimited. The limit is re-read on every wake, so
// SetMaxConcurrency reaches waiting tasks without recreating the dispatcher.
func (d *taskDispatcher) acquireSlot() func() {
d.mu.Lock()
defer d.mu.Unlock()
for d.limit > 0 && d.active >= d.limit {
d.slotsFree.Wait()
}
d.active++
return func() {
d.mu.Lock()
d.active--
d.mu.Unlock()
d.slotsFree.Broadcast()
}
}
func (d *taskDispatcher) Submit(
@@ -1069,10 +1122,8 @@ func (d *taskDispatcher) Submit(
if previous != nil {
<-previous
}
if d.slots != nil {
d.slots <- struct{}{}
defer func() { <-d.slots }()
}
releaseSlot := d.acquireSlot()
defer releaseSlot()
if done != nil {
defer func() {
close(done)
+1 -1
View File
@@ -334,7 +334,7 @@ func TestProcessStopRunningTransitionsToParked(t *testing.T) {
pool, err := archiecontainer.NewPool(ctx, archiecontainer.Config{
Image: "archie-agent:test",
DockerClient: dockerClient,
}, d.Log)
}, nil, d.Log)
if err != nil {
t.Fatal(err)
}
@@ -0,0 +1,103 @@
package daemon
import (
"context"
"log/slog"
"testing"
"time"
"github.com/samcharles93/archie-core/internal/config"
"github.com/samcharles93/archie-core/internal/domain/workflow"
)
// TestDaemonResizeTaskDispatcherTracksConfig pins the daemon's live-apply
// wrapper: the dispatcher is built from the running config and a later resize
// reaches it without rebuilding the daemon.
func TestDaemonResizeTaskDispatcherTracksConfig(t *testing.T) {
t.Parallel()
d := &Daemon{
Cfg: config.NewHolder(config.Config{Containers: config.ContainerConfig{MaxConcurrency: 2}}),
Log: slog.New(slog.DiscardHandler),
}
dispatcher := d.taskDispatcher()
dispatcher.mu.Lock()
booted := dispatcher.limit
dispatcher.mu.Unlock()
if booted != 2 {
t.Fatalf("dispatcher limit = %d, want the running config's 2", booted)
}
d.ResizeTaskDispatcher(5)
dispatcher.mu.Lock()
resized := dispatcher.limit
dispatcher.mu.Unlock()
if resized != 5 {
t.Fatalf("dispatcher limit after resize = %d, want 5", resized)
}
}
// TestTaskDispatcherResizesWithoutRestart pins the live half of
// archie-core-zfb0.2: the running dispatcher accepts a new
// containers.max_concurrency. Raising the limit admits queued work
// immediately; lowering it leaves running tasks holding their slots and
// admits new work only as they finish.
func TestTaskDispatcherResizesWithoutRestart(t *testing.T) {
t.Parallel()
dispatcher := newTaskDispatcher(1, nil)
releaseA := make(chan struct{})
aStarted := make(chan struct{})
bStarted := make(chan struct{})
releaseB := make(chan struct{})
cStarted := make(chan struct{})
dispatcher.Submit(context.Background(), &workflow.Task{Owner: "acme", Repo: "a"}, func(context.Context, *workflow.Task) {
close(aStarted)
<-releaseA
})
<-aStarted
dispatcher.Submit(context.Background(), &workflow.Task{Owner: "acme", Repo: "b"}, func(context.Context, *workflow.Task) {
close(bStarted)
<-releaseB
})
select {
case <-bStarted:
t.Fatal("second task started above max_concurrency 1")
case <-time.After(50 * time.Millisecond):
}
// Raising the limit must admit the queued task without a restart.
dispatcher.SetMaxConcurrency(2)
select {
case <-bStarted:
case <-time.After(time.Second):
t.Fatal("raising the limit did not admit the queued task")
}
// Lowering the limit applies as running tasks finish: a new task waits
// while two tasks hold slots, and starts only once they have drained
// below the new limit.
dispatcher.SetMaxConcurrency(1)
dispatcher.Submit(context.Background(), &workflow.Task{Owner: "acme", Repo: "c"}, func(context.Context, *workflow.Task) {
close(cStarted)
})
close(releaseA)
select {
case <-cStarted:
t.Fatal("a task started while two tasks still held slots under a lowered limit")
case <-time.After(50 * time.Millisecond):
}
close(releaseB)
select {
case <-cStarted:
case <-time.After(time.Second):
t.Fatal("lowered limit did not admit new work as running tasks finished")
}
dispatcher.Wait()
}
+1 -1
View File
@@ -357,7 +357,7 @@ func TestProcessCleansWorktreeOnTerminalNoChange(t *testing.T) {
pool, err := archiecontainer.NewPool(ctx, archiecontainer.Config{
Image: "archie-agent:test",
DockerClient: dockerClient,
}, d.Log)
}, nil, d.Log)
if err != nil {
t.Fatal(err)
}