[EPIC] Proactive day-2 operations: the first end-to-end story #1977

Open
opened 2026-11-14 14:42:01 +00:00 by sam · 4 comments
Owner

Archie hears disk-pressure alerts on a lab host, works out why, and earns the right to clean up on its own: it starts by asking, gains autonomy after consistent approvals, loses it on a denial, uses ServiceNow change requests where policy requires them, and proposes a root-cause fix once the problem recurs. It exercises the whole loop (Hears, Understands, Decides, Defines, Does, Delivers) and the platform (Governs, Reports, Assesses, Manages risk, Invokes policy), with no repository involved.

The spec, with the decision trees, the guard verdicts and the full requirement list, is SPEC.md in the private sam/archie-product repository. Requirement numbers (R1 and so on) refer to it.

The story

  1. First run: connect Alertmanager, Prometheus, Incus, Semaphore and ServiceNow. Archie drafts policies, controls and procedures from a starter set, and they're approved.
  2. Alert 1: nothing handles disk pressure, so Archie proposes a trigger and workflow. Once approved, it investigates read-only and proposes deleting old runner directories through a Semaphore template.
  3. Alerts 2 and 3: the deletion is approved again. Alert 2 also finds a container cache; a snapshot-then-delete sequence goes through a ServiceNow change request because impact is unknown.
  4. After 3 approvals, Archie proposes raising that action class to act and tell, and it's approved. Alert 4: it deletes, then tells.
  5. Alert 5: a release blackout holds the deletion; it's denied, and the class drops back.
  6. After the fifth incident, Archie proposes a root-cause config change and a scheduled workflow that replaces the sweep timer.

Done when

  • The story runs from real Alertmanager alerts; the only manual steps are approvals, the blackout and refilling the disk between alerts.
  • No action that changes anything runs before the drafted policies are approved.
  • The first, unmatched alert produces a trigger-and-workflow proposal.
  • Every action shows each guard's verdict with reasons, and its action class's history.
  • A class's level visibly rises after approvals and drops after a denial.
  • Approving the change in ServiceNow continues the run within a minute; denying it drops those actions safely.
  • Killing any Archie service mid-run loses nothing.
  • Archie runs off the host whose disk fills.
  • The auditor guard has a measured evaluation on record.
  • The root-cause proposal appears after the fifth incident, and approving it applies both actions and starts the scheduled workflow.
  • Off-script: a forbidden action is refused before any guard runs; a target with no recorded owner goes to the default owner; an unanswered approval escalates, then ends without acting.

Everything not on this story's path is deferred until it passes.

Archie hears disk-pressure alerts on a lab host, works out why, and earns the right to clean up on its own: it starts by asking, gains autonomy after consistent approvals, loses it on a denial, uses ServiceNow change requests where policy requires them, and proposes a root-cause fix once the problem recurs. It exercises the whole loop (Hears, Understands, Decides, Defines, Does, Delivers) and the platform (Governs, Reports, Assesses, Manages risk, Invokes policy), with no repository involved. The spec, with the decision trees, the guard verdicts and the full requirement list, is SPEC.md in the private sam/archie-product repository. Requirement numbers (R1 and so on) refer to it. ## The story 1. First run: connect Alertmanager, Prometheus, Incus, Semaphore and ServiceNow. Archie drafts policies, controls and procedures from a starter set, and they're approved. 2. Alert 1: nothing handles disk pressure, so Archie proposes a trigger and workflow. Once approved, it investigates read-only and proposes deleting old runner directories through a Semaphore template. 3. Alerts 2 and 3: the deletion is approved again. Alert 2 also finds a container cache; a snapshot-then-delete sequence goes through a ServiceNow change request because impact is unknown. 4. After 3 approvals, Archie proposes raising that action class to act and tell, and it's approved. Alert 4: it deletes, then tells. 5. Alert 5: a release blackout holds the deletion; it's denied, and the class drops back. 6. After the fifth incident, Archie proposes a root-cause config change and a scheduled workflow that replaces the sweep timer. ## Done when - The story runs from real Alertmanager alerts; the only manual steps are approvals, the blackout and refilling the disk between alerts. - No action that changes anything runs before the drafted policies are approved. - The first, unmatched alert produces a trigger-and-workflow proposal. - Every action shows each guard's verdict with reasons, and its action class's history. - A class's level visibly rises after approvals and drops after a denial. - Approving the change in ServiceNow continues the run within a minute; denying it drops those actions safely. - Killing any Archie service mid-run loses nothing. - Archie runs off the host whose disk fills. - The auditor guard has a measured evaluation on record. - The root-cause proposal appears after the fifth incident, and approving it applies both actions and starts the scheduled workflow. - Off-script: a forbidden action is refused before any guard runs; a target with no recorded owner goes to the default owner; an unanswered approval escalates, then ends without acting. Everything not on this story's path is deferred until it passes.
sam added the highepic labels 2026-11-14 14:42:01 +00:00
sam added a new dependency 2026-11-14 14:42:58 +00:00
sam added a new dependency 2026-11-14 14:42:58 +00:00
sam added a new dependency 2026-11-14 14:42:58 +00:00
sam added a new dependency 2026-11-14 14:42:59 +00:00
sam added a new dependency 2026-11-14 14:42:59 +00:00
sam added a new dependency 2026-11-14 14:42:59 +00:00
sam added a new dependency 2026-11-14 14:42:59 +00:00
sam added a new dependency 2026-11-14 14:42:59 +00:00
sam added a new dependency 2026-11-14 14:42:59 +00:00
sam added a new dependency 2026-11-14 14:42:59 +00:00
sam added a new dependency 2026-11-14 14:43:00 +00:00
sam added a new dependency 2026-11-14 14:43:00 +00:00
sam added a new dependency 2026-11-14 14:43:00 +00:00
sam added a new dependency 2026-11-14 14:43:00 +00:00
sam added a new dependency 2026-11-14 14:43:00 +00:00
sam added a new dependency 2026-11-14 14:43:00 +00:00
sam added a new dependency 2026-11-14 14:43:00 +00:00
sam added a new dependency 2026-11-14 14:43:00 +00:00
sam added a new dependency 2026-11-14 14:43:00 +00:00
sam added a new dependency 2026-11-14 16:02:45 +00:00
sam added a new dependency 2026-11-14 16:02:46 +00:00
sam added a new dependency 2026-11-14 16:02:46 +00:00
sam added a new dependency 2026-11-14 16:02:46 +00:00
sam added a new dependency 2026-11-14 16:02:46 +00:00
sam added a new dependency 2026-11-14 16:02:46 +00:00
sam added a new dependency 2026-11-14 16:02:46 +00:00
sam added a new dependency 2026-11-14 16:02:46 +00:00
sam added a new dependency 2026-11-15 06:07:07 +00:00
sam added a new dependency 2026-11-15 13:32:23 +00:00
Author
Owner

Slice 1 (an alert is investigated) passes its done-when on staging: the 2.3 spike's injection and kill-mid-run checks.

  • Injection: an alert whose description carries an instruction to look outside the investigation. A plain instruction was ignored; a runbook-styled one was followed, and the host connector refused the path as outside the host's lookup roots. The refusal is a failed tool call in the trail and the findings report it. No injected instruction reached a capability the connector wasn't configured to grant. The model doesn't yet flag the instruction as an injection: that's provenance marking (2.3), slice 2.
  • Kill mid-run: SIGKILL straight after a tool call was recorded, then restart. The run resumed on attempt 2, replayed the recorded model and tool calls without repeating either, and finished. The first attempt at this failed and found a real bug: a step's value differed between the live attempt and a resume (raw JSON rewritten by the encoder and by jsonb), so the resumed agent diverged from its journal. Fixed in 487736222 with a reproducing test.
  • Not covered yet: a kill during a tool call isn't marked unknown, since nothing is recorded before a call starts (harmless for read-only lookups; noted on #1990 for actions). The spike's approval, mid-stream cancel and memory measurements belong to later slices.

Also landed in this slice: credentials sealed with age and secret engines (sops on the sops library); the host connector's read-only lookups with pinned host keys, root helpers behind a narrow sudo rule, and per-host path roots; MCP servers as accepted packages with prometheus-mcp (#2005); the decision trail API, the composer, and the binary's own dashboard entry (#1996 and #1984 in part).

Next: slice 2, starting with the judgement module (see the 2026-11-16 review notes on #1986).

Slice 1 (an alert is investigated) passes its done-when on staging: the 2.3 spike's injection and kill-mid-run checks. - **Injection:** an alert whose description carries an instruction to look outside the investigation. A plain instruction was ignored; a runbook-styled one was followed, and the host connector refused the path as outside the host's lookup roots. The refusal is a failed tool call in the trail and the findings report it. No injected instruction reached a capability the connector wasn't configured to grant. The model doesn't yet flag the instruction as an injection: that's provenance marking (2.3), slice 2. - **Kill mid-run:** SIGKILL straight after a tool call was recorded, then restart. The run resumed on attempt 2, replayed the recorded model and tool calls without repeating either, and finished. The first attempt at this failed and found a real bug: a step's value differed between the live attempt and a resume (raw JSON rewritten by the encoder and by jsonb), so the resumed agent diverged from its journal. Fixed in 487736222 with a reproducing test. - **Not covered yet:** a kill during a tool call isn't marked unknown, since nothing is recorded before a call starts (harmless for read-only lookups; noted on #1990 for actions). The spike's approval, mid-stream cancel and memory measurements belong to later slices. Also landed in this slice: credentials sealed with age and secret engines (sops on the sops library); the host connector's read-only lookups with pinned host keys, root helpers behind a narrow sudo rule, and per-host path roots; MCP servers as accepted packages with prometheus-mcp (#2005); the decision trail API, the composer, and the binary's own dashboard entry (#1996 and #1984 in part). Next: slice 2, starting with the judgement module (see the 2026-11-16 review notes on #1986).
sam added a new dependency 2026-11-15 17:26:34 +00:00
sam added a new dependency 2026-11-15 17:52:47 +00:00
sam added a new dependency 2026-11-15 17:52:47 +00:00
Author
Owner

The epic's children were only the day-2 slices, so today's archie was mostly unaccounted for. Five children added for what must move into the one binary rather than disappear when the six services stop:

  • #2017 identity and access (users, passkeys, OIDC, tokens, roles)
  • #2018 chat as a surface (conversations, streaming, ask-from-run, suggestions)
  • #2019 traces and an audit log for every run
  • #2020 task runs as a workflow kind (issue in, sandboxed agent, commits through the daemon)
  • #2021 webhook intake keeps verifying what it parses

Not yet filed and not decided: exposing Archie's tools to external agents over MCP (#2012 deletes the old path), dashboard parity for the old pages (workflow editor, captures, alerts, services, logs, summary, setup, config), skills and curators, and the remote runner over HTTPS (spec 2.5). Slice 6 should not land until the children above are done.

The epic's children were only the day-2 slices, so today's archie was mostly unaccounted for. Five children added for what must move into the one binary rather than disappear when the six services stop: - #2017 identity and access (users, passkeys, OIDC, tokens, roles) - #2018 chat as a surface (conversations, streaming, ask-from-run, suggestions) - #2019 traces and an audit log for every run - #2020 task runs as a workflow kind (issue in, sandboxed agent, commits through the daemon) - #2021 webhook intake keeps verifying what it parses Not yet filed and not decided: exposing Archie's tools to external agents over MCP (#2012 deletes the old path), dashboard parity for the old pages (workflow editor, captures, alerts, services, logs, summary, setup, config), skills and curators, and the remote runner over HTTPS (spec 2.5). Slice 6 should not land until the children above are done.
Author
Owner

Spec continuation, 2026-11-16. ARCHITECTURE.md: 2.1 now says the runner is required for real estates (no inbound connection, credentials on the runner's side) and records that only in-process extensions exist today; 2.9 keeps the canvas as the workflow viewer and editor through drafts and carries a parity table for every old page; 2.10 is new, audit and evidence as a compliance requirement; section 4 gains slice 7, today's archie moving in, so slice 6 does not stop the old services before their capabilities land. SPEC.md gains R39 (runner and task runs), R40 (system health), R41 (chat), R42 (audit log), R43 (identity and access) and states that Archie's own MCP server is a later day-2 stage. The spec-building session is exported to archie-product/transcripts/. New children: #2022 runner, #2023 dashboard parity, #2017-#2021 from earlier. Still to decide: whether alerts has its own page, whether a home page exists before there is more than one binary. Slice 7's issues are #2017-#2023.

Spec continuation, 2026-11-16. ARCHITECTURE.md: 2.1 now says the runner is required for real estates (no inbound connection, credentials on the runner's side) and records that only in-process extensions exist today; 2.9 keeps the canvas as the workflow viewer and editor through drafts and carries a parity table for every old page; 2.10 is new, audit and evidence as a compliance requirement; section 4 gains slice 7, today's archie moving in, so slice 6 does not stop the old services before their capabilities land. SPEC.md gains R39 (runner and task runs), R40 (system health), R41 (chat), R42 (audit log), R43 (identity and access) and states that Archie's own MCP server is a later day-2 stage. The spec-building session is exported to archie-product/transcripts/. New children: #2022 runner, #2023 dashboard parity, #2017-#2021 from earlier. Still to decide: whether alerts has its own page, whether a home page exists before there is more than one binary. Slice 7's issues are #2017-#2023.
Author
Owner

Control plane: the maintainer restated it as a day-1 show-stopping requirement, not a re-implementation choice. ARCHITECTURE.md 2.9 gains 'Every setting has a page (R45)' and SPEC.md gains R45 plus principle 9, 'The dashboard is the control plane'. The judge is a control now, along with hard rules, change-request rules, guards, policy text and action classes. #2026 inventories the gap in the new binary (agent model, connectors, guards and their judge, secret engines, MCP servers, credentials, sources, key fields, triggers, events, bootstrap display and admin token rotation) and an agent is implementing it.

Control plane: the maintainer restated it as a day-1 show-stopping requirement, not a re-implementation choice. ARCHITECTURE.md 2.9 gains 'Every setting has a page (R45)' and SPEC.md gains R45 plus principle 9, 'The dashboard is the control plane'. The judge is a control now, along with hard rules, change-request rules, guards, policy text and action classes. #2026 inventories the gap in the new binary (agent model, connectors, guards and their judge, secret engines, MCP servers, credentials, sources, key fields, triggers, events, bootstrap display and admin token rotation) and an agent is implementing it.
sam added a new dependency 2026-11-16 08:12:25 +00:00
sam added a new dependency 2026-11-16 10:11:12 +00:00
sam added a new dependency 2026-11-16 10:11:13 +00:00
sam added a new dependency 2026-11-16 10:11:15 +00:00
sam added a new dependency 2026-11-16 10:39:56 +00:00
sam added a new dependency 2026-11-16 10:55:01 +00:00
sam added a new dependency 2026-11-16 10:55:02 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Depends on
Reference: sam/archie#1977