diff --git a/.github/skills/technical-prose-cleanup/SKILL.md b/.github/skills/technical-prose-cleanup/SKILL.md new file mode 100644 index 000000000..f19cae8f5 --- /dev/null +++ b/.github/skills/technical-prose-cleanup/SKILL.md @@ -0,0 +1,76 @@ +--- +name: technical-prose-cleanup +description: Use when asked to clean up awkward or machine-generated technical writing in Archie's modules without changing behaviour. Reviews one module at a time. +--- + +# Technical prose cleanup + +Follow `AGENTS.md`. Improve wording without changing behaviour, contracts or +technical meaning. Use Australian English. + +## Set the scope + +1. Check the working tree for existing changes. Inspect `modules/` without + editing and choose the smallest module with enough clear problems to justify + a pass, unless the maintainer has specified one. +2. Set the complete file scope before editing: exactly one module, at most + 15 files. Do not expand into other modules or start another batch. +3. Use subagents only when independent work warrants it. Assign at most three + in parallel, each with a disjoint list of owned files, this writing standard + and relevant examples. Instruct them to inspect the implementation, leave + clear prose alone, report ambiguities and make no commits or pushes. + The coordinator owns integration and validation. In a shared worktree, + avoid broad staging, resets, checkouts and formatters that rewrite other work. + +## Edit in context + +Check comments, errors, logs, test names, schema descriptions, CLI help, +prompts, user-facing strings and module documentation. Look for unnatural word +order, awkward possessives, unusual verbs, needless synonyms and wording that +obscures the operation. Read the surrounding implementation and tests before +rewriting. Do not mechanically replace words or edit text that already reads well. + +Use conventional terms, even when they repeat. Keep established idioms such as +“pin a dependency”, “drain a stream” and “acquire a lock”. Prefer precision over +brevity when conditions, timing, ownership or guarantees matter. + +| Instead of | Write | +| --- | --- | +| “PrepareRequest names the release to stage and who hears the outcome.” | “PrepareRequest specifies the release to stage and notification recipients.” | +| “ReadOnly leaves the agent only the tools that change nothing.” | “ReadOnly restricts the agent to read-only tools.” | +| “a release another key signed” | “a release signed by a different key” | +| “it recalls earlier incidents and files what it found” | “it retrieves previous incidents and stores new findings” | + +Comments explain why. Remove narration that adds nothing, but keep +non-obvious constraints, invariants and decisions. Retain required Go +documentation. Don't shorten a comment about replay into a generic retry +description: say what is recorded and what is rebuilt or resolved at runtime. +Likewise, distinguish a per-action reapproval limit from a run-wide limit, +and escalation within one request from a new request. + +Do not refactor or rename identifiers, configuration keys or API fields. +Do not edit generated files by hand, add tooling or dependencies, or reformat +unrelated code. Check consumers and tests before changing errors, logs or other +output. Keep machine-readable values and exact text used as a contract. If a +safe edit needs changes outside the scope or the meaning is unclear, leave +the text alone and report why. + +For prompts, preserve instructions, conditions, tool semantics and output +requirements. Check what the agent actually sees. For example, judgement runs +inside an action tool call; it is not a separate tool the agent calls directly. +The investigation prompt says: “If a tool call returns a judgement decision, +report the decision and any reasons it gives.” Keep matching fixtures in sync +when editing prompt text. + +## Review and stop + +Review the combined diff against both the implementation and the starting +working tree. Verify the module and file limits, check that unrelated edits +are intact, and confirm that no logic or contract changed. If implementation +work already shares a file, keep it out of the prose diff; separate by hunk +if asked to commit both. + +Run the relevant checks on the final tree and follow `AGENTS.md` for the +commit gate. Report failed or unavailable checks; do not fix unrelated +failures in this batch. Summarise the module, file count, main edits, +deliberately unchanged ambiguities and validation. Stop after this batch. diff --git a/CLAUDE.md b/CLAUDE.md index 7576f2eab..d55294984 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -131,8 +131,13 @@ When sources disagree, the higher one wins: - Log through the service's injected `*slog.Logger` with the `...Context` methods, so lines carry the OpenTelemetry trace id. Never the global `slog`. - If a fix fails twice, throw the block away and rewrite it simply. -- Comments say why, briefly. No migration history, phase names or issue - narration in code. +- Comments explain why, not what the code already says. No migration + history, phase names or issue narration in code. +- Write natural, idiomatic technical English in comments, errors, logs, + test names, descriptions, prompts and user-facing text. Use conventional + terminology and normal word order. Avoid invented metaphors, awkward + possessives, inverted sentences and clever paraphrases. Prefer repeating + an established term over inventing a synonym. - Generated files (protobuf, sqlc) are regenerated and committed with the change that caused them, never hand-edited. diff --git a/README.md b/README.md index 5414d74fb..3eb2d07cc 100644 --- a/README.md +++ b/README.md @@ -150,6 +150,10 @@ another [deployment template](deployments/README.md). , using [`deployments/dev.toml`](deployments/dev.toml). `task check` is the quality gate; [AGENTS.md](AGENTS.md) lists its tools. +| Project skill | Use | +| --- | --- | +| [`/technical-prose-cleanup`](.github/skills/technical-prose-cleanup/SKILL.md) | Review technical writing in one module without changing behaviour. | + ## Documentation - [Your first playbook](docs/guides/first-playbook.md) diff --git a/Taskfile.yml b/Taskfile.yml index b54bef4fc..2ce92adef 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -211,6 +211,30 @@ tasks: - task: dev:wait-state - go run ./cmd/archie-messaging -config-overlay deployments/dev.toml + dev:archie: + desc: Run the single archie binary against the development PostgreSQL (admin token "dev") + deps: [dev:postgres] + vars: + DIR: tmp/dev-archie + cmds: + - | + set -eu + mkdir -p {{.DIR}} + # Generated once: the age identity seals the dev credentials, so keep + # it across runs. Roles come from dev:postgres. + if [ ! -f {{.DIR}}/config.toml ]; then + umask 077 + { + echo 'listen = "127.0.0.1:8080"' + echo "admin_token_sha256 = \"$(printf dev | sha256sum | cut -d' ' -f1)\"" + echo "age_identity = \"$(go run ./cmd/archie -new-age-identity)\"" + echo '[database]' + echo 'url = "postgres://archie_runtime:archie@127.0.0.1:5432/archie?sslmode=disable"' + echo 'owner_url = "postgres://archie_migrator:archie@127.0.0.1:5432/archie?sslmode=disable"' + } > {{.DIR}}/config.toml + fi + - go run ./cmd/archie -config {{.DIR}}/config.toml + dev:ui: desc: Run Vite (requires the dashboard backend) dir: ui diff --git a/WORKFLOWS.md b/WORKFLOWS.md index e397bdaf6..d5ef412bb 100644 --- a/WORKFLOWS.md +++ b/WORKFLOWS.md @@ -133,7 +133,7 @@ _runs without a repository._ | `protect_tests` | bool | Write-block the repository test files. | | `read_only` | bool | Restrict the agent to read-only tools. | | `result` | mapping | A JSON Schema object the agent must return; later steps read it as steps..result. | -| `model` | text | The model alias that runs the agent, e.g. default or fast. Empty means default. | +| `model` | text | A model alias configured under Settings > Agent models, such as `fast`. Omit it or use `default` to select the organisation's default agent model. | ### `command.run` diff --git a/api/openapi.yaml b/api/openapi.yaml index fd6855aed..71bdd506b 100644 --- a/api/openapi.yaml +++ b/api/openapi.yaml @@ -2445,7 +2445,7 @@ paths: schema: $ref: "#/components/schemas/ErrorModel" description: Error - summary: Replay one of the group's events through the draft, changing nothing + summary: Replay a stored event against the draft workflow and policy in a dry run tags: - Drafts /api/drafts/{id}/publish: @@ -3108,6 +3108,14 @@ paths: /api/settings/agent-model: get: operationId: get-agent-model + parameters: + - description: Agent model alias; omit for the default model + explode: false + in: query + name: alias + schema: + description: Agent model alias; omit for the default model + type: string responses: "200": content: @@ -3126,6 +3134,14 @@ paths: - Governance put: operationId: put-agent-model + parameters: + - description: Agent model alias; omit for the default model + explode: false + in: query + name: alias + schema: + description: Agent model alias; omit for the default model + type: string requestBody: content: application/json: @@ -3173,6 +3189,27 @@ paths: summary: Ask a model one question before saving it tags: - Governance + /api/settings/agent-models: + get: + operationId: list-agent-models + responses: + "200": + content: + application/json: + schema: + additionalProperties: + $ref: "#/components/schemas/GovernanceModel" + type: object + description: OK + default: + content: + application/problem+json: + schema: + $ref: "#/components/schemas/ErrorModel" + description: Error + summary: List agent models by alias + tags: + - Governance /api/sources: get: operationId: list-sources diff --git a/e2e/README.md b/e2e/README.md index 2ca639d1a..56e08a4ac 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -48,6 +48,27 @@ The first run needs `pnpm install --dir e2e` and `pnpm --dir e2e exec playwright install chromium`. In the cloud environment, Chromium and Playwright are preinstalled; run with `NODE_PATH=$(pnpm root -g)`. +## Single-binary smoke check + +Point this check at an already running, disposable `archie` instance with its +own PostgreSQL database and no external delivery connectors. Set +`ARCHIE_SMOKE_TOKEN` to that instance's bootstrap token and +`ARCHIE_SMOKE_URL` to its IPv4 loopback URL (for example +`http://127.0.0.1:8080`), then run: + +```bash +node e2e/archie-smoke.cjs +``` + +The check saves a workflow and a one-minute schedule, then verifies that its +run links back to the schedule event and appears in the composer. It disables +the schedule after the check but leaves records for inspection. No model or +forge is needed. The legacy `task e2e:up` stack is not the target of this check. +If Archie is running in WSL and its loopback URL is not reachable from +Windows, run the check with Node in the same Linux environment as Archie. +If the schedule starts but the run does not finish, check the `events` and +`runs` logs; readiness alone does not verify that either worker can claim work. + ## Release evidence ```bash diff --git a/e2e/archie-smoke.cjs b/e2e/archie-smoke.cjs new file mode 100644 index 000000000..88639e18c --- /dev/null +++ b/e2e/archie-smoke.cjs @@ -0,0 +1,74 @@ +const assert = require("node:assert/strict"); +const { randomUUID } = require("node:crypto"); +const { setTimeout: delay } = require("node:timers/promises"); + +async function main() { + assert.ok(process.env.ARCHIE_SMOKE_URL, "Set ARCHIE_SMOKE_URL to a disposable Archie stack"); + const origin = new URL(process.env.ARCHIE_SMOKE_URL); + const token = process.env.ARCHIE_SMOKE_TOKEN; + assert.equal(origin.protocol, "http:"); + assert.equal(origin.hostname, "127.0.0.1", "Use a disposable Archie stack on IPv4 loopback"); + assert.ok(token, "Set ARCHIE_SMOKE_TOKEN to the disposable stack's admin token"); + + async function request(path, { method = "GET", body, authenticated = true, status = 200 } = {}) { + const response = await fetch(new URL(path, origin), { + method, + redirect: "error", + signal: AbortSignal.timeout(10_000), + headers: { + ...(authenticated && { Authorization: `Bearer ${token}` }), + ...(body !== undefined && { "Content-Type": "application/json" }), + }, + body: body === undefined ? undefined : JSON.stringify(body), + }).catch((err) => { + throw new Error(`${method} ${path}: could not reach Archie at ${origin.origin}: ${err.cause?.message || err.message}`, { cause: err }); + }); + assert.equal(response.status, status, `${method} ${path}: expected HTTP ${status}, got ${response.status}`); + return response.json(); + } + + const ready = await request("/readyz", { authenticated: false }); + assert.equal(ready.ready, true, "Archie is not ready"); + await request("/api/workflows", { authenticated: false, status: 401 }); + + const name = `smoke-${randomUUID()}`; + const saved = await request(`/api/workflows/${name}`, { + method: "PUT", + body: { body: `repository: none\nsteps:\n - id: notify\n type: run.notify\n settings: {title: ${name}}\n` }, + }); + const trigger = { workflow: name, every: "1m" }; + try { + await request(`/api/triggers/${name}`, { method: "PUT", body: trigger }); + const deadline = Date.now() + 90_000; + let last = "no scheduled run"; + while (Date.now() < deadline) { + const recent = await request("/api/runs"); + const run = recent.find((row) => row.workflow === name); + if (run) { + last = `run ${run.id}: ${run.state}`; + assert.ok(!["failed", "cancelled"].includes(run.state), `${last}: ${run.error || "no error returned"}`); + if (run.state === "succeeded") { + const trail = await request(`/api/runs/${run.id}`); + assert.equal(trail.run.workflow_version, saved.version, "Run used the wrong workflow version"); + assert.equal(trail.origin?.trigger, name, "Run trail is missing the schedule trigger"); + assert.ok(trail.origin?.event_id, "Run trail is missing the originating event"); + assert.equal(trail.origin.type, "schedule.tick"); + const messages = await request("/api/composer/messages"); + assert.equal(messages.filter((m) => m.run_id === run.id && m.title === name).length, 1, + "Expected one composer notification linked to the run"); + console.log(`Schedule, run trail and composer notification verified for ${run.id}`); + return; + } + } + await delay(1_000); + } + throw new Error(`Timed out waiting for ${name}: ${last}`); + } finally { + await request(`/api/triggers/${name}`, { method: "PUT", body: { ...trigger, disabled: true } }); + } +} + +main().catch((err) => { + console.error(err.message); + process.exitCode = 1; +}); diff --git a/go.mod b/go.mod index 219294e98..6074ca63e 100644 --- a/go.mod +++ b/go.mod @@ -13,7 +13,6 @@ require ( github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/cloudflare/circl v1.6.5 // indirect github.com/containerd/errdefs/pkg v0.3.0 // indirect - github.com/danielgtaylor/huma/v2 v2.40.0 // indirect github.com/distribution/reference v0.6.0 // indirect github.com/docker/go-connections v0.8.2 // indirect github.com/docker/go-units v0.5.0 // indirect @@ -72,6 +71,7 @@ require ( github.com/cedar-policy/cedar-go v1.9.1 github.com/containerd/errdefs v1.0.0 github.com/coreos/go-oidc/v3 v3.21.0 + github.com/danielgtaylor/huma/v2 v2.40.0 github.com/go-git/go-billy/v6 v6.0.0-beta.1 github.com/go-git/go-git/v6 v6.0.0-beta.1 github.com/go-jose/go-jose/v4 v4.1.9 diff --git a/go.sum b/go.sum index 98126b6b6..8c00a21c9 100644 --- a/go.sum +++ b/go.sum @@ -70,6 +70,8 @@ github.com/fxamacker/cbor/v2 v2.10.0 h1:HVF3T5vYwLqO81ebPirWEPNv2SMrUMD3HLnKgEr6 github.com/fxamacker/cbor/v2 v2.10.0/go.mod h1:uSwlToFuT9BZAKRxxVm3qQC8DR4tlWl7D6pf20iRYH4= github.com/gliderlabs/ssh v0.3.8 h1:a4YXD1V7xMF9g5nTkdfnja3Sxy1PVDCj1Zg4Wb8vY6c= github.com/gliderlabs/ssh v0.3.8/go.mod h1:xYoytBv1sV0aL3CavoDuJIQNURXkkfPA/wxQ1pL1fAU= +github.com/go-chi/chi/v5 v5.5.0 h1:xdKQ1dJmVJsmgahrXSKhUx8NyVPzpwdlyrrQFpMPJgg= +github.com/go-chi/chi/v5 v5.5.0/go.mod h1:d1IG7GtKPSjPyBP365PH8sAGVHk3qfB/IhjFNRH1wWM= github.com/go-git/gcfg/v2 v2.0.2 h1:MY5SIIfTGGEMhdA7d7JePuVVxtKL7Hp+ApGDJAJ7dpo= github.com/go-git/gcfg/v2 v2.0.2/go.mod h1:/lv2NsxvhepuMrldsFilrgct6pxzpGdSRC13ydTLSLs= github.com/go-git/go-billy/v6 v6.0.0-beta.1 h1:OJ4r207zcEbeq/DJNajQUOMd+pph3r9A7hRq1Sjl1bA= diff --git a/internal/domain/workflow/control_steps_test.go b/internal/domain/workflow/control_steps_test.go index 6fc84b581..61ede175e 100644 --- a/internal/domain/workflow/control_steps_test.go +++ b/internal/domain/workflow/control_steps_test.go @@ -9,6 +9,23 @@ import ( "github.com/samcharles93/archie-core/internal/domain/workflow/task" ) +// TestMessageSendWithoutRepository guards workflows that only send a message +// from being rejected by the worktree requirement. +func TestMessageSendWithoutRepository(t *testing.T) { + for _, mode := range []string{"none", "optional"} { + t.Run(mode, func(t *testing.T) { + _, err := ParseDefinition("id: notify\nrepository: "+mode+` +steps: + - type: message.send + settings: {channel: telegram, chat_id: "123", text: "Done"} +`, StepRegistry{MessageSendStepName: newMessageSendStage}) + if err != nil { + t.Fatal(err) + } + }) + } +} + func TestValidateControlTargets(t *testing.T) { registry := StepRegistry{ AgentRunStepName: newAgentRunStage, diff --git a/internal/domain/workflow/definition.go b/internal/domain/workflow/definition.go index 4fa1787b2..ac18b85be 100644 --- a/internal/domain/workflow/definition.go +++ b/internal/domain/workflow/definition.go @@ -37,6 +37,7 @@ type YAMLDefinition struct { var repoFreeStepTypes = map[string]bool{ AgentRunStepName: true, WorkflowCallStepName: true, FinishStepName: true, HandoffStepName: true, ApproveStepName: true, CloseIssueStepName: true, CommentStepName: true, + MessageSendStepName: true, } // NeedsRepository reports whether a step type can run only with a worktree. diff --git a/internal/platform/pg/migrations/00036_agent_model_aliases.sql b/internal/platform/pg/migrations/00036_agent_model_aliases.sql new file mode 100644 index 000000000..905b7365c --- /dev/null +++ b/internal/platform/pg/migrations/00036_agent_model_aliases.sql @@ -0,0 +1,8 @@ +-- +goose Up +ALTER TABLE governance.model_roles DROP CONSTRAINT model_roles_role_check; +ALTER TABLE governance.model_roles ADD CONSTRAINT model_roles_role_check + CHECK (role = 'agent' OR (role <> 'agent:default' AND role ~ '^agent:[a-zA-Z0-9][a-zA-Z0-9_.-]{0,62}$')); + +-- +goose Down +ALTER TABLE governance.model_roles DROP CONSTRAINT model_roles_role_check; +ALTER TABLE governance.model_roles ADD CONSTRAINT model_roles_role_check CHECK (role = 'agent'); diff --git a/modules/approvals/approvals.go b/modules/approvals/approvals.go index 00c8f3b52..a2c5f626a 100644 --- a/modules/approvals/approvals.go +++ b/modules/approvals/approvals.go @@ -1,11 +1,7 @@ -// Package approvals owns an approval request: one judged action waiting for a -// person's answer, and the answer that settles it. A request is made when -// judgement holds an action at propose and wait, and an answer is an event in: -// it wakes the run waiting on that key, which then runs the action only if the -// answer named the exact action that was judged. -// -// The request binds to the decision and its digest, so an approval covers the -// action that was shown and nothing else. +// Package approvals manages requests for human approval and records responses. +// A response to an action request resumes the waiting run. The request binds +// the approval to the decision ID and action digest so it cannot authorize a +// different action. package approvals import ( @@ -33,8 +29,8 @@ import ( "github.com/samcharles93/archie-core/modules/judgement" ) -// The states a request can be in. A settled request never changes again, -// except that an approval the action no longer matched is discarded. +// A settled request cannot be answered again. An approved request may still +// be discarded before the approval is counted. const ( StatePending = "pending" StateApproved = "approved" @@ -72,12 +68,12 @@ const ( KindRaise = "raise" ) -// Request is one judged action waiting for a person's answer. +// Request tracks approval of an action or a proposed increase in autonomy. type Request struct { ID uuid.UUID Kind string RunID uuid.UUID - // Key is the run's wait key, stored so an answer knows what to wake. + // Key identifies the run wait that receives the response. Key string Decision judgement.Decision DecisionID uuid.UUID @@ -89,11 +85,9 @@ type Request struct { Level string Summary string State string - // Deadline is when the request runs out for good: the end of the last - // escalation step. + // Deadline applies to the whole request, not just the current escalation step. Deadline time.Time - // Step is the escalation step the request is at, and StepDeadline when it - // moves on. + // Step and StepDeadline track the current escalation step and its deadline. Step int StepDeadline time.Time // ChangeRequest is whether the request went through a change request @@ -104,7 +98,7 @@ type Request struct { AnsweredVia string } -// Answer is what the run wakes with. +// Answer records the decision and who made it. type Answer struct { RequestID uuid.UUID `json:"request_id"` Approved bool `json:"approved"` @@ -113,12 +107,13 @@ type Answer struct { At time.Time `json:"at"` } -// Approval is the authority this answer gives judgement to run the action. +// Approval binds the approver's identity and response time to the decision ID +// and action digest. func (a Answer) Approval(d judgement.Decision) judgement.Approval { return judgement.Approval{DecisionID: d.ID, Digest: d.Action.Digest, By: a.By, At: a.At} } -// View is a request as the API shows it. +// View is the API representation of an approval request. type View struct { ID uuid.UUID `json:"id"` Kind string `json:"kind"` @@ -129,11 +124,10 @@ type View struct { Summary string `json:"summary"` State string `json:"state"` Deadline time.Time `json:"deadline"` - // Step is the escalation step the request is at, and StepDeadline when it - // moves to the next. + // Step and StepDeadline describe the current escalation step. Step int `json:"step"` StepDeadline time.Time `json:"step_deadline,omitzero"` - // DecisionID is the judged action the request is about. + // DecisionID identifies the judgement decision or, for a raise request, the proposed raise. DecisionID uuid.UUID `json:"decision_id"` RequestedAt time.Time `json:"requested_at"` AnsweredBy string `json:"answered_by,omitzero"` @@ -141,7 +135,7 @@ type View struct { AnsweredAt time.Time `json:"answered_at,omitzero"` } -// View is r as the API shows it. +// View converts a request to its API representation. func (r Request) View() View { return View{ ID: r.ID, Kind: r.Kind, RunID: r.RunID, Action: r.Connector + " " + r.Action, @@ -157,16 +151,15 @@ type Signaller interface { Signal(ctx context.Context, id uuid.UUID, key string, payload jsontext.Value) error } -// Delivery carries a request to every channel someone can answer on. +// Delivery sends approval requests and publishes responses and notifications. type Delivery interface { Ask(ctx context.Context, a deliveries.ApprovalRequest) error Told(ctx context.Context, a deliveries.ApprovalRequest, answer string) error Notify(ctx context.Context, n deliveries.Notification) error } -// Classes is where answers become autonomy: an approval counts toward raising -// an action class, a denial tightens it, and a raise a person approves -// loosens it. +// Classes updates action-class autonomy. Denials reduce autonomy; approvals +// count towards an increase that a person must approve separately. type Classes interface { Approved(ctx context.Context, decisionID uuid.UUID, by string, need int, deadline time.Time) (*judgement.Raise, error) Denied(ctx context.Context, decisionID uuid.UUID) error @@ -174,14 +167,13 @@ type Classes interface { CancelRaise(ctx context.Context, id uuid.UUID) error } -// Policy is what the organisation decided about approvals. +// Policy provides the organisation's deadline and approval threshold for raises. type Policy interface { ApprovalDeadline(ctx context.Context) (time.Duration, error) RaiseAfter(ctx context.Context) (int, error) } -// People says who an identity belongs to. An answer from an identity it -// doesn't verify is refused. +// People resolves verified channel identities to people. type People interface { Verify(ctx context.Context, identity string) (governance.Verified, bool, error) } @@ -200,10 +192,7 @@ type Module struct { now func() time.Time } -// NewModule returns the approvals module for orgID. Answers are delivered by -// tell and wake the run through wake, and are counted into classes under -// policy. An answer a channel delivers is accepted only from an identity -// people verifies. +// NewModule creates the approvals module for orgID. func NewModule(db *sqlx.DB, log *slog.Logger, orgID string, tell Delivery, wake Signaller, classes Classes, policy Policy, people People) *Module { return &Module{ orgID: orgID, store: store.New(db), tell: tell, wake: wake, classes: classes, policy: policy, people: people, @@ -211,12 +200,10 @@ func NewModule(db *sqlx.DB, log *slog.Logger, orgID string, tell Delivery, wake } } -// Ask records a request for d and puts it to the first step of path. It runs -// out for good when the last step does. Asking again for the same run and key -// returns the request that stands, so a replayed step asks once. An action -// judgement's change-request rules pick goes through a change request when a -// change-request connector is installed, and every other one through the -// approvers' channels. +// Ask reuses the request for the same run and key so retries do not create +// duplicate requests. It uses a change-request connector when judgement +// requires a change request and a connector is configured. Otherwise, it +// sends the request to the approval channels with the first step's deadline. func (m *Module) Ask(ctx context.Context, runID uuid.UUID, key string, d judgement.Decision, path []governance.PathStep) (Request, error) { if len(path) == 0 { return Request{}, ErrNoPath @@ -267,9 +254,6 @@ func (m *Module) Ask(ctx context.Context, runID uuid.UUID, key string, d judgeme return r, nil } -// answerAs settles request id with a verified person's decision and wakes the run -// waiting on it. Only a pending request settles: a second answer, or one that -// arrives after the deadline, is refused and wakes nobody. func (m *Module) answerAs(ctx context.Context, id uuid.UUID, by, via string, approved bool) (Request, error) { now := m.now().UTC() if _, err := m.pending(ctx, id, now); err != nil { @@ -296,9 +280,8 @@ func (m *Module) answerAs(ctx context.Context, id uuid.UUID, by, via string, app if err := m.wakeWith(ctx, row, answer); err != nil { return Request{}, err } - // The run has what it waited for. A denial tightens the class now. An - // approval counts only once the action is about to run on it, so one that - // is discarded first counts for nothing. + // Count denials immediately. Defer counting approvals until execution + // is imminent so discarded approvals do not increase autonomy. if !answer.Approved { if err := m.classes.Denied(ctx, row.DecisionID); err != nil { return Request{}, err @@ -307,8 +290,6 @@ func (m *Module) answerAs(ctx context.Context, id uuid.UUID, by, via string, app return requestOf(row, judgement.Decision{}), nil } -// learn counts an approval of an action into its class. An approval that -// completes a run of them earns a raise, which is put to a person. func (m *Module) learn(ctx context.Context, row store.Request, a Answer) error { need, err := m.policy.RaiseAfter(ctx) if err != nil { @@ -325,8 +306,6 @@ func (m *Module) learn(ctx context.Context, row store.Request, a Answer) error { return m.askRaise(ctx, raise) } -// askRaise puts a raise to a person on every channel. If it can't be asked the -// raise is withdrawn, so the class proposes it again at its next approval. func (m *Module) askRaise(ctx context.Context, r *judgement.Raise) error { class := r.Class row, err := m.store.Put(ctx, store.Request{ @@ -343,6 +322,7 @@ func (m *Module) askRaise(ctx context.Context, r *judgement.Raise) error { }) } if err != nil { + // Cancel the raise so a later approval can trigger another proposal. if cerr := m.classes.CancelRaise(ctx, r.ID); cerr != nil { err = errors.Join(err, cerr) } @@ -359,19 +339,17 @@ func (m *Module) answerRaise(ctx context.Context, row store.Request, a Answer) e } text := answerText(a) if a.Approved && !raised { - text += " The class had changed since, so its level stays." + text += " This approval did not increase the action class's autonomy level." } return m.tell.Told(ctx, deliveries.ApprovalRequest{ RequestID: row.ID, Key: row.Key + ":ask", Title: "Raise answered", }, text) } -// raiseSummary is what a person is shown about a raise: the class, the change -// and the approvals that earned it. func raiseSummary(r *judgement.Raise) string { c := r.Class var s strings.Builder - fmt.Fprintf(&s, "%s: %s on %s in %s. Raise from %s to %s.\nEarned by %d approvals in a row:", + fmt.Fprintf(&s, "%s: %s on %s in %s. Proposed autonomy increase: %s to %s.\nConsecutive approvals (%d):", c.Connector, c.Action, c.TargetKind, c.Environment, r.From, r.To, len(r.Evidence)) for _, e := range r.Evidence { fmt.Fprintf(&s, "\n- %s, %s", e.By, e.At.UTC().Format("2006-01-02 15:04 UTC")) @@ -379,13 +357,10 @@ func raiseSummary(r *judgement.Raise) string { return s.String() } -// Settle answers a request from an answer a channel delivered as an event, -// given by identity (such as "telegram:1001"). An answer from an identity no -// person holds, or from a person who may not answer that kind of request, is -// refused and the refusal recorded with its reason. An answer that can't -// settle the request, because it was already answered, has expired or names no -// request, also reports false rather than an error: retrying the event would -// never change that. +// Settle processes a response from a channel identity such as "telegram:1001". +// It records why it rejected an unverified or unauthorized response. It returns +// false without an error for recorded rejections and for requests that are +// missing, expired or already answered, since retrying won't help. func (m *Module) Settle(ctx context.Context, id uuid.UUID, identity, via string, approved bool) (bool, error) { row, err := m.store.ByID(ctx, m.orgID, id) if errors.Is(err, ErrNotFound) { @@ -414,11 +389,6 @@ func (m *Module) Settle(ctx context.Context, id uuid.UUID, identity, via string, return false, err } -// verify says whether identity may answer request row. An action is answered -// by someone who answers for actions and whom the request has reached on its -// escalation path, or who holds authority, and an action at the escalate level -// needs authority. A raise changes policy, so only a person with authority -// answers it. func (m *Module) verify(ctx context.Context, identity string, row store.Request) (governance.Verified, error) { who, ok, err := m.people.Verify(ctx, identity) if err != nil { @@ -446,7 +416,6 @@ func mayAnswer(who governance.Verified, row store.Request) error { return nil } -// askedOf is the ids of the people the request has reached. func askedOf(row store.Request) []string { var ids []string if err := json.Unmarshal([]byte(row.Asked), &ids); err != nil { @@ -455,16 +424,14 @@ func askedOf(row store.Request) []string { return ids } -// refusal reports whether err is a refused answer rather than a failure. +// refusal distinguishes a rejected response from an operational error. func refusal(err error) bool { return errors.Is(err, ErrUnverified) || errors.Is(err, ErrNotApprover) || errors.Is(err, ErrNoAuthority) || errors.Is(err, ErrNotAsked) || errors.Is(err, ErrNeedsAuthority) } -// Expire settles a request whose last escalation step ran out, so the trail -// shows why it never ran. Nothing is acted on, an open change request is -// withdrawn, and the report goes to every channel. Expiry says nothing about -// the action's class, so it is not tightened. +// Expire records a timeout without reducing the action class's autonomy: +// receiving no response is not a denial. func (m *Module) Expire(ctx context.Context, id uuid.UUID) error { row, err := m.store.Answer(ctx, m.orgID, id, StateExpired, "deadline", "clock", m.now().UTC()) if err != nil { @@ -486,16 +453,15 @@ func (m *Module) Expire(ctx context.Context, id uuid.UUID) error { } if err := m.tell.Told(ctx, deliveries.ApprovalRequest{ RunID: row.RunID, RequestID: row.ID, Key: row.Key + ":ask", - Title: "Approval expired", Body: "The request ran out of time, so nothing was run." + withdrawn, + Title: "Approval expired", Body: "The approval request expired without a response. The action did not run." + withdrawn, }, "Expired without an answer."); err != nil { return err } - // The report goes to every channel. Its key is stable, so an Expire - // retried after a failed delivery sends it once. + // Use a stable key so retrying the expiry does not duplicate the notification. return m.tell.Notify(ctx, deliveries.Notification{ RunID: row.RunID, Key: row.Key + ":expired", Title: "Approval timed out: " + row.Connector + " " + row.Action, - Body: "Nobody answered before the last escalation step ran out, so " + row.Connector + " " + row.Action + + Body: "No one responded before the final escalation deadline, so " + row.Connector + " " + row.Action + " did not run." + withdrawn + " " + row.Summary, }) } @@ -526,8 +492,7 @@ func (m *Module) ForRun(ctx context.Context, runID uuid.UUID) ([]Request, error) return out, nil } -// Outcomes says what became of the requests made for the decisions in ids, -// for the history of the action class they belong to. +// Outcomes returns approval outcomes by decision ID for the action class history. func (m *Module) Outcomes(ctx context.Context, ids []uuid.UUID) (map[uuid.UUID]judgement.Outcome, error) { rows, err := m.store.ByDecisions(ctx, m.orgID, ids) if err != nil { @@ -560,9 +525,8 @@ func (m *Module) Register(api huma.API) { }, m.answer) } -// bootstrapAdmin names the identity the bootstrap admin token stands for. The -// token guard has already authenticated it, so it counts as a person with -// authority and skips the people check until real sign-in exists. +// The token guard authenticates the bootstrap admin before the API handler, +// so the handler does not look up a channel identity. const bootstrapAdmin = "admin" type answerInput struct { @@ -604,9 +568,6 @@ func (m *Module) answer(ctx context.Context, in *answerInput) (*httpapi.Reply[Vi return httpapi.OK(out.View()), nil } -// pending reads a request that is still open and inside its deadline. The -// store settles only a pending row; this refuses the answer before that, with -// the reason. func (m *Module) pending(ctx context.Context, id uuid.UUID, now time.Time) (store.Request, error) { row, err := m.store.ByID(ctx, m.orgID, id) if err != nil { @@ -618,8 +579,6 @@ func (m *Module) pending(ctx context.Context, id uuid.UUID, now time.Time) (stor return row, nil } -// settleable says whether an answer now may settle r. A request settles once: -// a second answer, and one that arrives after the deadline, changes nothing. func settleable(r store.Request, now time.Time) error { switch { case r.State != StatePending: @@ -652,8 +611,7 @@ func answerText(a Answer) string { return fmt.Sprintf("%s by %s via %s.", verb, a.By, a.Via) } -// Summary is what a person is shown about the action they are asked to allow: -// what it does, to what, and what held it back. +// Summary shows the approver the action, its target and why it requires approval. func Summary(d judgement.Decision) string { s := fmt.Sprintf("%s: %s on %s %q.", d.Action.Connector, d.Action.Name, d.Action.Target.Kind, d.Action.Target.ID) if d.Action.Effect.Summary != "" { @@ -682,7 +640,7 @@ func Summary(d judgement.Decision) string { } // inputLine renders an action's input as "key=value" pairs in key order, or -// the raw JSON when it isn't a flat object. +// the raw JSON when it is not a nonempty object. func inputLine(input jsontext.Value) string { var fields map[string]jsontext.Value if err := json.Unmarshal(input, &fields); err != nil || len(fields) == 0 { @@ -700,7 +658,6 @@ func inputLine(input jsontext.Value) string { return strings.Join(parts, ", ") } -// requestOf is a stored row as the module's Request. func requestOf(row store.Request, d judgement.Decision) Request { r := Request{ ID: row.ID, Kind: row.Kind, RunID: row.RunID, Key: row.Key, Decision: d, DecisionID: row.DecisionID, RequestedAt: row.CreatedAt, diff --git a/modules/approvals/approvals_test.go b/modules/approvals/approvals_test.go index d1c1a00be..cefc07314 100644 --- a/modules/approvals/approvals_test.go +++ b/modules/approvals/approvals_test.go @@ -19,11 +19,9 @@ func (f fakePeople) Verify(_ context.Context, identity string) (governance.Verif return p, ok, nil } -// TestVerifyAcceptsAnAnswerOnlyFromAPersonWhoMayGiveIt guards the rule that an -// approval counts only from a verified identity, on every channel: an identity -// no person holds, a person without the standing for that kind of request, a -// person the request hasn't reached on its escalation path, or one without the -// authority the action's level needs, is refused rather than settling it. +// TestVerifyAcceptsAnAnswerOnlyFromAPersonWhoMayGiveIt checks that a verified +// identity alone is not enough: the request kind, escalation step and required +// authority also determine who can respond. func TestVerifyAcceptsAnAnswerOnlyFromAPersonWhoMayGiveIt(t *testing.T) { people := fakePeople{ "telegram:1001": {ID: "owner", Name: "Owner", Approver: true}, @@ -41,18 +39,18 @@ func TestVerifyAcceptsAnAnswerOnlyFromAPersonWhoMayGiveIt(t *testing.T) { row store.Request wantErr error }{ - {name: "an approver the request has reached answers", identity: "telegram:1001", row: action(propose, "owner")}, + {name: "an approver can respond once notified", identity: "telegram:1001", row: action(propose, "owner")}, {name: "an unmapped identity is refused", identity: "telegram:9999", row: action(propose, "owner"), wantErr: ErrUnverified}, {name: "an unmapped identity can't answer a raise", identity: "telegram:9999", row: store.Request{Kind: KindRaise}, wantErr: ErrUnverified}, - {name: "a person who isn't an approver is refused an action", identity: "telegram:1003", row: action(propose, "owner"), wantErr: ErrNotApprover}, - {name: "the target's owner answers when the request was put to them", identity: "telegram:1003", row: action(propose, "bystander")}, - {name: "an approver without authority is refused a raise", identity: "telegram:1001", row: store.Request{Kind: KindRaise}, wantErr: ErrNoAuthority}, - {name: "authority answers a raise", identity: "telegram:1002", row: store.Request{Kind: KindRaise}}, - {name: "a person on the path is refused until the request reaches them", identity: "telegram:1004", row: action(propose, "owner"), wantErr: ErrNotAsked}, - {name: "a person on the path answers once the request reaches them", identity: "telegram:1004", row: action(propose, "owner", "second")}, - {name: "authority answers before it is asked", identity: "telegram:1002", row: action(propose, "owner")}, - {name: "an action at escalate needs authority even from someone asked", identity: "telegram:1001", row: action(escalate, "owner"), wantErr: ErrNeedsAuthority}, - {name: "authority answers an action at escalate", identity: "telegram:1002", row: action(escalate, "owner")}, + {name: "an unlisted person who isn't an approver cannot respond", identity: "telegram:1003", row: action(propose, "owner"), wantErr: ErrNotApprover}, + {name: "the target owner can respond when explicitly asked", identity: "telegram:1003", row: action(propose, "bystander")}, + {name: "an approver without authority cannot respond to a raise", identity: "telegram:1001", row: store.Request{Kind: KindRaise}, wantErr: ErrNoAuthority}, + {name: "a person with authority can respond to a raise", identity: "telegram:1002", row: store.Request{Kind: KindRaise}}, + {name: "a later approver cannot respond before escalation", identity: "telegram:1004", row: action(propose, "owner"), wantErr: ErrNotAsked}, + {name: "a later approver can respond after escalation", identity: "telegram:1004", row: action(propose, "owner", "second")}, + {name: "a person with authority can respond without being asked", identity: "telegram:1002", row: action(propose, "owner")}, + {name: "the escalate level requires authority even when the approver was asked", identity: "telegram:1001", row: action(escalate, "owner"), wantErr: ErrNeedsAuthority}, + {name: "a person with authority can respond at the escalate level", identity: "telegram:1002", row: action(escalate, "owner")}, } m := &Module{people: people} for _, tt := range tests { @@ -68,9 +66,8 @@ func TestVerifyAcceptsAnAnswerOnlyFromAPersonWhoMayGiveIt(t *testing.T) { } } -// TestSettleableRefusesASecondAnswerOrALateOne guards the rule that makes an -// approval one decision: the first answer settles a request, so a replaying -// answer or one that arrives after the deadline wakes no run and runs nothing. +// TestSettleableRefusesASecondAnswerOrALateOne checks that a settled request or +// one at its deadline cannot accept a response. func TestSettleableRefusesASecondAnswerOrALateOne(t *testing.T) { deadline := time.Date(2026, 11, 16, 12, 0, 0, 0, time.UTC) request := func(state string) store.Request { @@ -82,8 +79,8 @@ func TestSettleableRefusesASecondAnswerOrALateOne(t *testing.T) { now time.Time wantErr error }{ - {name: "a pending request settles inside its deadline", row: request(StatePending), now: deadline.Add(-time.Minute)}, - {name: "a pending request is too late after its deadline", row: request(StatePending), now: deadline, wantErr: ErrExpired}, + {name: "a pending request accepts a response before the deadline", row: request(StatePending), now: deadline.Add(-time.Minute)}, + {name: "a pending request rejects a response at the deadline", row: request(StatePending), now: deadline, wantErr: ErrExpired}, {name: "an approved request doesn't settle again", row: request(StateApproved), now: deadline.Add(-time.Minute), wantErr: ErrAnswered}, {name: "a denied request doesn't settle again", row: request(StateDenied), now: deadline.Add(-time.Minute), wantErr: ErrAnswered}, {name: "an expired request doesn't settle", row: request(StateExpired), now: deadline.Add(-time.Minute), wantErr: ErrAnswered}, diff --git a/modules/approvals/escalation.go b/modules/approvals/escalation.go index 7eca791f3..dfba7bb44 100644 --- a/modules/approvals/escalation.go +++ b/modules/approvals/escalation.go @@ -12,24 +12,20 @@ import ( "github.com/samcharles93/archie-core/modules/governance" ) -// ChangeRequests is a change-request connector. Judgement's change-request -// rules decide which held actions need one; this opens and withdraws them. +// ChangeRequests manages external change requests for actions that require them. type ChangeRequests interface { - // Open raises the change request for r. It may be called again for the same - // request and must not open a second one. + // Open must be idempotent for the same approval request. Open(ctx context.Context, r Request) error // Withdraw withdraws r's change request if it is still open. Withdraw(ctx context.Context, r Request) error } -// UseChangeRequests makes the module open the change requests judgement asks -// for through c. Without one, those approvals go to the approvers' channels, -// so a rule never strands a request where nobody can answer it. +// UseChangeRequests configures the change-request connector. Without one, +// approval requests still go to the approvers' channels. func (m *Module) UseChangeRequests(c ChangeRequests) { m.changes = c } -// Escalate passes a pending request to step n of its path, telling every -// channel who it has reached and until when. The step's deadline runs from -// now. A request that has since been answered is left alone. +// Escalate advances a pending request to step n. It sets the step's deadline +// relative to the time of escalation. func (m *Module) Escalate(ctx context.Context, id uuid.UUID, n int, to governance.PathStep) error { row, err := m.store.ByID(ctx, m.orgID, id) if err != nil { @@ -43,12 +39,12 @@ func (m *Module) Escalate(ctx context.Context, id uuid.UUID, n int, to governanc if to.Name != "" { who = to.Name } - // The notice goes first and is idempotent by key, so an escalation retried - // after the move failed still tells people once. + // Send the notice before updating the request. If the update fails, + // retrying with the same key won't duplicate the notice. if err := m.tell.Notify(ctx, deliveries.Notification{ RunID: row.RunID, Key: row.Key + ":escalated:" + strconv.Itoa(n), Title: "Approval escalated: " + row.Connector + " " + row.Action, - Body: "Nobody answered in time, so it now goes to " + who + " and waits until " + until.Format("2006-01-02 15:04 UTC") + ". " + row.Summary, + Body: "No one responded before the previous deadline. Next approver: " + who + ". Response deadline: " + until.Format("2006-01-02 15:04 UTC") + ". " + row.Summary, }); err != nil { return err } @@ -59,9 +55,8 @@ func (m *Module) Escalate(ctx context.Context, id uuid.UUID, n int, to governanc return nil } -// Accept counts an approval toward raising its action class, once, when the -// action it covers is about to run. An approval that is discarded first never -// counts. +// Accept counts an approval towards an increase in autonomy just before the +// action runs. Each approval counts at most once; discarded approvals do not count. func (m *Module) Accept(ctx context.Context, id uuid.UUID) error { row, counted, err := m.store.Count(ctx, m.orgID, id) if err != nil || !counted { @@ -72,9 +67,8 @@ func (m *Module) Accept(ctx context.Context, id uuid.UUID) error { }) } -// Discard withdraws an approval whose action changed before it could run, and -// says why. A discarded approval doesn't count toward a raise, and a request -// for the changed action is a new one. +// Discard records why an approval is no longer valid and notifies the approval +// channels. A discarded approval cannot count towards a raise. func (m *Module) Discard(ctx context.Context, id uuid.UUID, because string) error { row, discarded, err := m.store.Discard(ctx, m.orgID, id, because) if err != nil || !discarded { @@ -90,5 +84,5 @@ func (m *Module) Discard(ctx context.Context, id uuid.UUID, because string) erro return nil } -// ErrNoPath means a request was asked without anyone to put it to. +// ErrNoPath indicates that an approval request has no escalation path. var ErrNoPath = errors.New("an approval request needs an escalation path") diff --git a/modules/approvals/internal/store/store.go b/modules/approvals/internal/store/store.go index 14bd5fd24..871243393 100644 --- a/modules/approvals/internal/store/store.go +++ b/modules/approvals/internal/store/store.go @@ -1,5 +1,4 @@ -// Package store holds the approvals module's SQL: the requests waiting for an -// answer, and what was answered. +// Package store persists approval requests, responses and rejections. package store import ( @@ -13,19 +12,17 @@ import ( "github.com/jmoiron/sqlx" ) -// Request is one judged action waiting for a person's answer. +// Request is the stored record of an action or raise approval request. type Request struct { ID uuid.UUID `db:"id"` OrgID string `db:"org_id"` RunID uuid.UUID `db:"run_id"` Key string `db:"key"` - // DecisionID is the judged action the request is about, or for a raise - // request the raise. + // DecisionID identifies the judgement decision or the proposed raise. DecisionID uuid.UUID `db:"decision_id"` Kind string `db:"kind"` Digest string `db:"digest"` - // The action's own names, kept so a request reads back without its - // decision. + // Copy the action details so the request can be read without loading the decision. Connector string `db:"connector"` Action string `db:"action"` TargetKind string `db:"target_kind"` @@ -38,9 +35,8 @@ type Request struct { AnsweredAt sql.NullTime `db:"answered_at"` AnsweredBy sql.NullString `db:"answered_by"` AnsweredVia sql.NullString `db:"answered_via"` - // Step is the escalation step the request is at, StepDeadline when that - // step runs out, and Asked a JSON array of the ids of the people it has - // reached so far. + // Asked is a JSON array of person IDs reached so far, not just the + // person assigned to the current step. Step int `db:"step"` StepDeadline sql.NullTime `db:"step_deadline"` Asked string `db:"asked"` @@ -48,8 +44,7 @@ type Request struct { // rather than the approvers' channels. ChangeRequest bool `db:"change_request"` // Counted is whether the approval was counted toward raising its class. - Counted bool `db:"counted"` - // DiscardedBecause is why an approval no longer stood. + Counted bool `db:"counted"` DiscardedBecause sql.NullString `db:"discarded_because"` } @@ -65,8 +60,8 @@ const requestColumns = `id, org_id, run_id, key, decision_id, kind, digest, conn target_id, level, summary, state, deadline, created_at, answered_at, answered_by, answered_via, step, step_deadline, asked::text AS asked, change_request, counted, discarded_because` -// Put records r unless a request already exists for its run and key, and -// returns the request that stands either way. +// Put inserts r or returns the existing request for the same organisation, +// run and key. func (s *Store) Put(ctx context.Context, r Request) (Request, error) { var out Request err := s.db.GetContext(ctx, &out, ` @@ -107,8 +102,8 @@ func (s *Store) Get(ctx context.Context, orgID string, runID uuid.UUID, key stri return out, nil } -// Answer settles one request, and only while it is still pending. It returns -// the request as it now stands, or the one that already settled it. +// Answer updates only pending requests. If another response already settled +// the request, it returns the stored record unchanged. func (s *Store) Answer(ctx context.Context, orgID string, id uuid.UUID, state, by, via string, at time.Time) (Request, error) { var out Request err := s.db.GetContext(ctx, &out, ` @@ -125,9 +120,9 @@ func (s *Store) Answer(ctx context.Context, orgID string, id uuid.UUID, state, b return out, nil } -// Escalate moves a pending request to step, which has reached person and runs -// out at until. It returns false when the request is no longer pending, or has -// already moved that far, so a replayed escalation changes nothing. +// Escalate advances a pending request to step, sets the deadline to until and +// records person if supplied. It returns false if the request has already +// reached that step or is no longer pending. func (s *Store) Escalate(ctx context.Context, orgID string, id uuid.UUID, step int, person string, until time.Time) (Request, bool, error) { var out Request err := s.db.GetContext(ctx, &out, ` @@ -145,8 +140,8 @@ func (s *Store) Escalate(ctx context.Context, orgID string, id uuid.UUID, step i return out, true, nil } -// Discard withdraws an approval that no longer stands, and only while it has -// not been counted. It returns false when there was nothing to discard. +// Discard invalidates an approval only if it has not yet been counted. +// It returns false if no record was updated. func (s *Store) Discard(ctx context.Context, orgID string, id uuid.UUID, because string) (Request, bool, error) { var out Request err := s.db.GetContext(ctx, &out, ` @@ -162,8 +157,8 @@ func (s *Store) Discard(ctx context.Context, orgID string, id uuid.UUID, because return out, true, nil } -// Count marks an approval as counted, once. It returns false when the approval -// was already counted or no longer stands. +// Count marks a valid action approval as counted. It returns false if no +// eligible approval was updated. func (s *Store) Count(ctx context.Context, orgID string, id uuid.UUID) (Request, bool, error) { var out Request err := s.db.GetContext(ctx, &out, ` diff --git a/modules/drafts/drafts.go b/modules/drafts/drafts.go index 493e1d460..5462ef777 100644 --- a/modules/drafts/drafts.go +++ b/modules/drafts/drafts.go @@ -1,7 +1,6 @@ -// Package drafts holds what an unmatched group of events could become: the -// trigger it proposes, the workflow that trigger would start and any policy -// that workflow needs, with a plain-language summary and a dry run. Nothing in -// a draft is in force until it is published. +// Package drafts manages proposed triggers, workflows and policies for groups +// of unmatched events. Drafts can be tested by replaying a stored event in a +// dry run before the trigger is enabled. package drafts import ( @@ -28,11 +27,10 @@ import ( "github.com/samcharles93/archie-core/modules/runs" ) -// replayable is how many of a group's newest events a dry run can choose from. +// replayable limits dry runs to the group's most recently received events. const replayable = 50 -// Groups is where a draft finds the unmatched events it is made from, and -// where its trigger is saved. +// Groups provides unmatched event groups, replay tracking and trigger storage. type Groups interface { Group(ctx context.Context, id uuid.UUID) (events.Group, bool, error) GroupEvents(ctx context.Context, id uuid.UUID, limit int) ([]events.Stored, error) @@ -40,7 +38,7 @@ type Groups interface { SaveTrigger(ctx context.Context, t events.TriggerSpec) error } -// Workflows stores workflow documents and runs them dry. +// Workflows installs workflow documents and manages dry runs. type Workflows interface { Install(ctx context.Context, orgID, name, body, by string) (int, error) Has(ctx context.Context, orgID, name string) (bool, error) @@ -48,7 +46,8 @@ type Workflows interface { DryReport(ctx context.Context, orgID string, id uuid.UUID) (runs.DryReport, bool, error) } -// Policies checks a draft's policy against the set in force and adds it. +// Policies validates a draft's policy against the current policy set and +// adds it when the draft is published. type Policies interface { Check(ctx context.Context, kind, extra string) error Append(ctx context.Context, kind, extra, by string) (judgement.PolicySetView, error) @@ -90,7 +89,7 @@ func (m *Module) Register(api huma.API) { }, m.edit) huma.Register(api, huma.Operation{ OperationID: "dry-run-draft", Method: http.MethodPost, Path: "/api/drafts/{id}/dry-run", - Summary: "Replay one of the group's events through the draft, changing nothing", Tags: tags, + Summary: "Replay a stored event against the draft workflow and policy in a dry run", Tags: tags, }, m.dryRun) huma.Register(api, huma.Operation{ OperationID: "publish-draft", Method: http.MethodPost, Path: "/api/drafts/{id}/publish", @@ -102,7 +101,7 @@ type idInput struct { ID string `path:"id" format:"uuid"` } -// View is a draft as the API returns it. +// View is the API representation of a draft. type View struct { ID uuid.UUID `json:"id"` GroupID uuid.UUID `json:"group_id"` @@ -114,7 +113,7 @@ type View struct { // Policy is Cedar policy text added to the set in force when published. Policy string `json:"policy"` Summary string `json:"summary"` - // DryRun is the last dry run of the draft as it stands. + // DryRun reports the latest dry run since the draft was last edited. DryRun *runs.DryReport `json:"dry_run,omitzero"` DryEventID *uuid.UUID `json:"dry_event_id,omitzero"` PublishedAt *time.Time `json:"published_at,omitzero"` @@ -159,8 +158,6 @@ var ( policyKinds = map[string]string{judgement.KindHardRules: "hard rule", judgement.KindChangeRequest: "change request rule"} ) -// summarize says in plain language when a draft fires and what it does. It is -// built from the draft's own fields, so it can't disagree with them. func summarize(d store.Draft) string { var filter map[string]string _ = json.Unmarshal(d.Filter, &filter) @@ -197,7 +194,8 @@ func summarize(d store.Draft) string { return b.String() } -// defaultName is a trigger name that is valid and tells groups apart. +// defaultName uses part of the group ID to distinguish groups with the same +// source and event type. func defaultName(g events.Group) string { slug := strings.Trim(nameChars.ReplaceAllString(strings.ToLower(g.Source+"-"+g.EventType), "-"), "-") if len(slug) > 50 { @@ -222,12 +220,12 @@ func (m *Module) list(ctx context.Context, _ *struct{}) (*httpapi.Reply[[]View], return httpapi.OK(out), nil } -// create makes a group's draft, or returns the one it has. A new draft -// proposes the group's trigger and a copy of the investigate workflow. +// create reuses an existing group draft rather than replacing any edits +// with the default investigation workflow. func (m *Module) create(ctx context.Context, in *idInput) (*httpapi.Reply[View], error) { id, err := uuid.Parse(in.ID) if err != nil { - return nil, huma.Error400BadRequest("an id is a UUID") + return nil, huma.Error400BadRequest("ID must be a UUID") } g, found, err := m.groups.Group(ctx, id) switch { @@ -268,8 +266,6 @@ type editInput struct { } } -// edit replaces a draft's name, workflow and policy after checking them. What -// doesn't validate changes nothing. func (m *Module) edit(ctx context.Context, in *editInput) (*httpapi.Reply[View], error) { d, err := m.draft(ctx, in.ID) if err != nil { @@ -295,7 +291,6 @@ func (m *Module) edit(ctx context.Context, in *editInput) (*httpapi.Reply[View], return m.respond(ctx, d) } -// check validates what a draft would put in force. func (m *Module) check(ctx context.Context, name, document, policyKind, policy string) error { if !validName.MatchString(name) { return &runs.InvalidError{Problems: []runs.Problem{{Path: "name", Message: "lowercase letters, digits and hyphens, starting with a letter"}}} @@ -331,8 +326,8 @@ type dryRunInput struct { } } -// dryRun replays one of the group's stored events through the draft's workflow -// in dry mode: the newest by default. +// dryRun tests the draft workflow and policy against a stored event. If the +// caller omits the event ID, it replays the most recently received event. func (m *Module) dryRun(ctx context.Context, in *dryRunInput) (*httpapi.Reply[View], error) { d, err := m.draft(ctx, in.ID) if err != nil { @@ -372,7 +367,7 @@ func (m *Module) dryRun(ctx context.Context, in *dryRunInput) (*httpapi.Reply[Vi return m.respond(ctx, d) } -// pick returns the event asked for, or the newest when none is. +// pick expects events in newest-first order so a missing ID selects the newest. func pick(stored []events.Stored, id uuid.UUID) (events.Stored, bool) { for _, e := range stored { if id == uuid.Nil || e.ID == id { @@ -382,10 +377,10 @@ func pick(stored []events.Stored, id uuid.UUID) (events.Stored, bool) { return events.Stored{}, false } -// publish puts a draft in force: its policy is added to the set, its workflow -// is installed as the next version, and last its trigger is saved, so nothing -// fires before the rules that govern it exist. A retry after a failure carries -// on from where it stopped. +// publish installs the policy and workflow before saving the trigger so the +// trigger cannot fire before both are available. A recorded workflow version +// lets a retry skip the name-conflict check. The retry still calls the policy +// and workflow installers, which avoid duplicating unchanged content. func (m *Module) publish(ctx context.Context, in *idInput) (*httpapi.Reply[View], error) { d, err := m.draft(ctx, in.ID) if err != nil { @@ -438,7 +433,7 @@ func (m *Module) publish(ctx context.Context, in *idInput) (*httpapi.Reply[View] func (m *Module) draft(ctx context.Context, rawID string) (store.Draft, error) { id, err := uuid.Parse(rawID) if err != nil { - return store.Draft{}, huma.Error400BadRequest("an id is a UUID") + return store.Draft{}, huma.Error400BadRequest("ID must be a UUID") } d, err := m.store.ByID(ctx, m.orgID, id) switch { diff --git a/modules/drafts/internal/store/store.go b/modules/drafts/internal/store/store.go index cd4249d41..d0c3df3c5 100644 --- a/modules/drafts/internal/store/store.go +++ b/modules/drafts/internal/store/store.go @@ -17,7 +17,7 @@ import ( // ErrNotFound means no such draft. var ErrNotFound = errors.New("no such draft") -// ErrTaken means another draft already has that name or that group. +// ErrTaken indicates that another draft already uses the name. var ErrTaken = errors.New("name already used by another draft") // Draft is one stored draft. @@ -34,7 +34,8 @@ type Draft struct { Policy string `db:"policy"` DryRunID uuid.NullUUID `db:"dry_run_id"` DryEventID uuid.NullUUID `db:"dry_event_id"` - // InstalledVersion is the workflow version a publish installed, once one did. + // InstalledVersion is recorded before the trigger is saved so a retry + // can distinguish an earlier installation from a workflow name conflict. InstalledVersion sql.NullInt32 `db:"installed_version"` PublishedAt sql.NullTime `db:"published_at"` CreatedAt time.Time `db:"created_at"` @@ -52,8 +53,7 @@ type Store struct { // New returns a store on db. func New(db *sqlx.DB) *Store { return &Store{db: db} } -// Create stores d unless its group already has a draft, and returns the draft -// the group has. +// Create inserts a draft or returns the existing draft for the group. func (s *Store) Create(ctx context.Context, d Draft) (Draft, error) { _, err := s.db.ExecContext(ctx, ` INSERT INTO drafts.drafts (id, org_id, group_id, name, source, event_type, filter, document, policy_kind, policy) @@ -98,8 +98,8 @@ func (s *Store) List(ctx context.Context, orgID string) ([]Draft, error) { return out, nil } -// Edit replaces what a person edits of an unpublished draft. A dry run made of -// an earlier version no longer describes it, so it is cleared. +// Edit updates an unpublished draft and clears its dry-run references because +// the previous run no longer represents the current draft. func (s *Store) Edit(ctx context.Context, orgID string, id uuid.UUID, name, document, policyKind, policy string) error { res, err := s.db.ExecContext(ctx, ` UPDATE drafts.drafts @@ -118,7 +118,7 @@ func (s *Store) Edit(ctx context.Context, orgID string, id uuid.UUID, name, docu return nil } -// SetDryRun records the dry run last made of a draft and the event it replayed. +// SetDryRun records the latest dry run and the event it replayed. func (s *Store) SetDryRun(ctx context.Context, orgID string, id, runID, eventID uuid.UUID) error { if _, err := s.db.ExecContext(ctx, ` UPDATE drafts.drafts SET dry_run_id = $3, dry_event_id = $4, updated_at = now() @@ -128,7 +128,7 @@ func (s *Store) SetDryRun(ctx context.Context, orgID string, id, runID, eventID return nil } -// MarkInstalled records the workflow version a publish installed. +// MarkInstalled records the workflow version installed during publication. func (s *Store) MarkInstalled(ctx context.Context, orgID string, id uuid.UUID, version int) error { if _, err := s.db.ExecContext(ctx, ` UPDATE drafts.drafts SET installed_version = $3, updated_at = now() WHERE org_id = $1 AND id = $2`, @@ -138,7 +138,7 @@ func (s *Store) MarkInstalled(ctx context.Context, orgID string, id uuid.UUID, v return nil } -// MarkPublished records that a draft is in force. +// MarkPublished records the publication time without overwriting it on retry. func (s *Store) MarkPublished(ctx context.Context, orgID string, id uuid.UUID) error { if _, err := s.db.ExecContext(ctx, ` UPDATE drafts.drafts SET published_at = COALESCE(published_at, now()), updated_at = now() diff --git a/modules/events/internal/store/store.go b/modules/events/internal/store/store.go index 89189f6f7..565936b90 100644 --- a/modules/events/internal/store/store.go +++ b/modules/events/internal/store/store.go @@ -53,7 +53,8 @@ type Event struct { Own bool `db:"own"` OriginRunID uuid.NullUUID `db:"origin_run_id"` GroupKey sql.NullString `db:"group_key"` - GroupFilter jsontext.Value `db:"group_filter"` + // Scan into []byte to preserve SQL NULL for events with no group. + GroupFilter []byte `db:"group_filter"` } // Trigger starts a workflow for events of one type from one source whose diff --git a/modules/governance/governance.go b/modules/governance/governance.go index 12b998a15..20d2c1a94 100644 --- a/modules/governance/governance.go +++ b/modules/governance/governance.go @@ -13,7 +13,9 @@ import ( "log/slog" "net/http" "net/url" + "regexp" "slices" + "strings" "time" "github.com/danielgtaylor/huma/v2" @@ -29,7 +31,27 @@ import ( // is no built-in default. var ErrNoAgentModel = errors.New("no agent model is set for the organisation") -const roleAgent = "agent" +const ( + roleAgent = "agent" + agentAliasPrefix = "agent:" +) + +var agentAliasPattern = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,62}$`) + +// ValidateAgentModelAlias accepts an empty alias for the default model. +func ValidateAgentModelAlias(alias string) error { + if alias != "" && !agentAliasPattern.MatchString(alias) { + return fmt.Errorf("invalid agent model alias %q: use 1-63 letters, digits, dots, underscores or hyphens, starting with a letter or digit", alias) + } + return nil +} + +func agentModelRole(alias string) string { + if alias == "" || alias == "default" { + return roleAgent + } + return agentAliasPrefix + alias +} // DefaultApprovalDeadline is how long an approval request waits for an answer // when the organisation hasn't said. @@ -247,11 +269,23 @@ func (m *Module) UseCredentials(c Credentials) { m.credentials = c } // AgentModel returns the organisation's agent model. func (m *Module) AgentModel(ctx context.Context) (Model, error) { - role, ok, err := m.store.ModelRole(ctx, m.orgID, roleAgent) + return m.AgentModelForAlias(ctx, "") +} + +// AgentModelForAlias resolves a model without falling back for unknown aliases. +func (m *Module) AgentModelForAlias(ctx context.Context, alias string) (Model, error) { + if err := ValidateAgentModelAlias(alias); err != nil { + return Model{}, err + } + key := agentModelRole(alias) + role, ok, err := m.store.ModelRole(ctx, m.orgID, key) if err != nil { return Model{}, err } if !ok { + if key != roleAgent { + return Model{}, fmt.Errorf("agent model alias %q: %w", alias, ErrNoAgentModel) + } return Model{}, ErrNoAgentModel } return Model{Class: role.Class, BaseURL: role.BaseURL, Model: role.Model, APIKeyCredential: role.APIKeyCredential}, nil @@ -355,6 +389,10 @@ func (m *Module) Register(api huma.API) { OperationID: "put-settings", Method: http.MethodPut, Path: "/api/settings", Summary: "Change the organisation settings", Tags: tags, }, m.putSettings) + huma.Register(api, huma.Operation{ + OperationID: "list-agent-models", Method: http.MethodGet, Path: "/api/settings/agent-models", + Summary: "List agent models by alias", Tags: tags, + }, m.listAgentModels) huma.Register(api, huma.Operation{ OperationID: "get-agent-model", Method: http.MethodGet, Path: "/api/settings/agent-model", Summary: "Get the agent model", Tags: tags, @@ -495,8 +533,32 @@ func (s Settings) knows(people []store.Person) error { return nil } -func (m *Module) getAgentModel(ctx context.Context, _ *struct{}) (*httpapi.Reply[Model], error) { - model, err := m.AgentModel(ctx) +func (m *Module) listAgentModels(ctx context.Context, _ *struct{}) (*httpapi.Reply[map[string]Model], error) { + rows, err := m.store.AgentModelRoles(ctx, m.orgID) + if err != nil { + m.log.ErrorContext(ctx, "list agent models", "error", err) + return nil, huma.NewError(http.StatusInternalServerError, "couldn't list agent models") + } + out := make(map[string]Model, len(rows)) + for _, row := range rows { + alias := strings.TrimPrefix(row.Role, agentAliasPrefix) + if row.Role == roleAgent { + alias = "default" + } + out[alias] = Model{Class: row.Class, BaseURL: row.BaseURL, Model: row.Model, APIKeyCredential: row.APIKeyCredential} + } + return httpapi.OK(out), nil +} + +type agentModelAlias struct { + Alias string `query:"alias" doc:"Agent model alias; omit for the default model"` +} + +func (m *Module) getAgentModel(ctx context.Context, in *agentModelAlias) (*httpapi.Reply[Model], error) { + if err := ValidateAgentModelAlias(in.Alias); err != nil { + return nil, huma.NewError(http.StatusBadRequest, err.Error()) + } + model, err := m.AgentModelForAlias(ctx, in.Alias) switch { case errors.Is(err, ErrNoAgentModel): return nil, huma.NewError(http.StatusNotFound, err.Error()) @@ -508,9 +570,16 @@ func (m *Module) getAgentModel(ctx context.Context, _ *struct{}) (*httpapi.Reply } } -// putAgentModel sets the agent model. An empty base URL takes the class's -// published default, and the stored value is always explicit. -func (m *Module) putAgentModel(ctx context.Context, in *struct{ Body Model }) (*httpapi.Reply[Model], error) { +type agentModelInput struct { + Alias string `query:"alias" doc:"Agent model alias; omit for the default model"` + Body Model +} + +// putAgentModel stores the resolved base URL so saved settings are explicit. +func (m *Module) putAgentModel(ctx context.Context, in *agentModelInput) (*httpapi.Reply[Model], error) { + if err := ValidateAgentModelAlias(in.Alias); err != nil { + return nil, huma.NewError(http.StatusBadRequest, err.Error()) + } req := in.Body model, err := settle(req) if err != nil { @@ -520,7 +589,7 @@ func (m *Module) putAgentModel(ctx context.Context, in *struct{ Body Model }) (* return nil, huma.NewError(http.StatusBadRequest, err.Error()) } if err := m.store.PutModelRole(ctx, store.ModelRole{ - OrgID: m.orgID, Role: roleAgent, Class: model.Class, BaseURL: model.BaseURL, Model: model.Model, + OrgID: m.orgID, Role: agentModelRole(in.Alias), Class: model.Class, BaseURL: model.BaseURL, Model: model.Model, APIKeyCredential: model.APIKeyCredential, }); err != nil { m.log.ErrorContext(ctx, "put agent model", "error", err) diff --git a/modules/governance/internal/store/store.go b/modules/governance/internal/store/store.go index dcf2064f0..baa3e40af 100644 --- a/modules/governance/internal/store/store.go +++ b/modules/governance/internal/store/store.go @@ -47,6 +47,17 @@ func (s *Store) ModelRole(ctx context.Context, orgID, role string) (ModelRole, b return out, true, nil } +func (s *Store) AgentModelRoles(ctx context.Context, orgID string) ([]ModelRole, error) { + var out []ModelRole + if err := s.db.SelectContext(ctx, &out, ` + SELECT org_id, role, class, base_url, model, api_key_credential + FROM governance.model_roles + WHERE org_id = $1 AND (role = 'agent' OR role LIKE 'agent:%')`, orgID); err != nil { + return nil, fmt.Errorf("list agent models: %w", err) + } + return out, nil +} + // PutModelRole records the organisation's model for a role. func (s *Store) PutModelRole(ctx context.Context, m ModelRole) error { if _, err := s.db.NamedExecContext(ctx, ` diff --git a/modules/memory/internal/store/store.go b/modules/memory/internal/store/store.go index 189c10a8d..9b7b08275 100644 --- a/modules/memory/internal/store/store.go +++ b/modules/memory/internal/store/store.go @@ -1,5 +1,4 @@ -// Package store holds the memory module's SQL: incidents and the proposals -// raised from them. +// Package store persists incident history and proposals for root-cause fixes. package store import ( @@ -11,7 +10,7 @@ import ( "github.com/jmoiron/sqlx" ) -// Incident is one investigation of one target as stored. +// Incident is the stored record of a run's investigation. type Incident struct { ID uuid.UUID `db:"id"` RunID uuid.UUID `db:"run_id"` @@ -24,7 +23,7 @@ type Incident struct { At time.Time `db:"at"` } -// Proposal is a root-cause fix asked for on a target. +// Proposal is a request to fix a recurring cause on a target. type Proposal struct { ID uuid.UUID `db:"id"` TargetKind string `db:"target_kind"` @@ -42,8 +41,8 @@ type Store struct { // New returns a store on db. func New(db *sqlx.DB) *Store { return &Store{db: db} } -// Upsert records the incident of a run, replacing what an earlier attempt of the -// same run recorded. +// Upsert inserts an incident or updates the target, cause and summary for +// the same organisation and run. It preserves the original event and timestamp. func (s *Store) Upsert(ctx context.Context, orgID string, in Incident) error { if _, err := s.db.ExecContext(ctx, ` INSERT INTO memory.incidents (id, org_id, run_id, target_kind, target_id, cause, summary, event) @@ -57,7 +56,7 @@ func (s *Store) Upsert(ctx context.Context, orgID string, in Incident) error { return nil } -// SetActions records what was done in a run's incident. +// SetActions replaces the recorded action outcomes for a run's incident. func (s *Store) SetActions(ctx context.Context, orgID string, runID uuid.UUID, actions []byte) error { if _, err := s.db.ExecContext(ctx, ` UPDATE memory.incidents SET actions = $3 WHERE org_id = $1 AND run_id = $2`, @@ -93,8 +92,8 @@ func (s *Store) ByTarget(ctx context.Context, orgID, kind, id string, limit int) return out, nil } -// Propose records p unless the target and cause already have one, and returns -// the one that stands. +// Propose stores one proposal per organisation, target and cause. If one +// already exists, it updates the incident count and returns the original text. func (s *Store) Propose(ctx context.Context, orgID string, p Proposal) (Proposal, error) { var out Proposal if err := s.db.GetContext(ctx, &out, ` diff --git a/modules/memory/memory.go b/modules/memory/memory.go index 1e47f100c..16026c1da 100644 --- a/modules/memory/memory.go +++ b/modules/memory/memory.go @@ -1,7 +1,6 @@ -// Package memory is what Archie remembers about a target: each incident -// investigated on it, the cause found, and what was done about it. A cause that -// keeps coming back on a target, whatever became of each attempt to handle it, -// raises a proposal for a root-cause fix. +// Package memory records investigated incidents by target, including their +// causes and action outcomes. When a cause reaches the recurrence threshold +// for a target, the module proposes a root-cause fix regardless of action outcomes. package memory import ( @@ -28,17 +27,17 @@ const maxHistory = 200 // MaxRecurrence is the largest recurrence threshold the history can confirm. const MaxRecurrence = 100 -// Target is what an incident happened to: a kind and an id, such as host web-1. +// Target identifies the affected resource by kind and ID, such as host web-1. type Target struct { Kind string `json:"kind"` ID string `json:"id"` } -// Action is something done during an incident and how it ended. +// Action records an action considered during an incident and its outcome. type Action struct { Action string `json:"action"` Target string `json:"target"` - // State is what became of it: ran, refused, or the approval's state. + // State records whether the action ran, was refused or has an approval status. State string `json:"state"` Detail string `json:"detail,omitzero"` } @@ -51,7 +50,7 @@ type Incident struct { Actions []Action `json:"actions"` } -// Line is the incident as one entry of a list an agent reads. +// Line formats the incident as one entry in the history shown to the agent. func (i Incident) Line() string { line := fmt.Sprintf("- %s, cause %q: %s", i.At.Format("2006-01-02"), i.Cause, i.Summary) if len(i.Actions) == 0 { @@ -68,17 +67,16 @@ func (i Incident) Line() string { return line + " Actions: " + strings.Join(parts, "; ") + "." } -// Review is what recording a run's actions found about its cause. +// Review reports how often the cause occurred and any proposal for a fix. type Review struct { - // Repeats is how many incidents on the target have this cause, this one - // included. + // Repeats counts matching incidents in the target's recent history, + // including the current incident if it is in that history. Repeats int - // Proposal is the root-cause fix asked for, empty until the cause has - // recurred enough. + // Proposal is the text of the request for a root-cause fix. It is empty + // below the recurrence threshold. Proposal string } -// ledger is where incidents and proposals are kept. type ledger interface { Upsert(ctx context.Context, orgID string, in store.Incident) error SetActions(ctx context.Context, orgID string, runID uuid.UUID, actions []byte) error @@ -87,7 +85,7 @@ type ledger interface { Propose(ctx context.Context, orgID string, p store.Proposal) (store.Proposal, error) } -// Module remembers one organisation's incidents. +// Module manages incident history and proposals for one organisation. type Module struct { orgID string ledger ledger @@ -100,14 +98,14 @@ func NewModule(db *sqlx.DB, orgID string) *Module { var unsafeCause = regexp.MustCompile(`[^a-z0-9]+`) -// Cause is a cause as it is compared: lower case words joined by hyphens, so -// "Disk pressure" and "disk-pressure" are one cause. +// Cause normalises labels for comparison: "Disk pressure" and "disk-pressure" +// both produce "disk-pressure". func Cause(s string) string { return strings.Trim(unsafeCause.ReplaceAllString(strings.ToLower(s), "-"), "-") } -// Record remembers what a run's investigation found: the event, the target it -// concerned and the cause. Recording again replaces the earlier attempt's. +// Record saves the incident for a run. Recording it again updates the target, +// cause and summary without replacing the original event. func (m *Module) Record(ctx context.Context, runID uuid.UUID, event jsontext.Value, t Target, cause, summary string) error { t = Target{Kind: strings.TrimSpace(t.Kind), ID: strings.TrimSpace(t.ID)} cause = Cause(cause) @@ -123,7 +121,7 @@ func (m *Module) Record(ctx context.Context, runID uuid.UUID, event jsontext.Val }) } -// Past returns the incidents recorded on t before the run's own, newest first. +// Past returns recent incidents for t, newest first, excluding the current run. func (m *Module) Past(ctx context.Context, runID uuid.UUID, t Target) ([]Incident, error) { rows, err := m.ledger.ByTarget(ctx, m.orgID, strings.TrimSpace(t.Kind), strings.TrimSpace(t.ID), maxHistory) if err != nil { @@ -143,10 +141,9 @@ func (m *Module) Past(ctx context.Context, runID uuid.UUID, t Target) ([]Inciden return out, nil } -// Finish remembers what was done in a run's incident and checks whether its -// cause has come back enough to ask for a root-cause fix: that is when -// threshold incidents on the target share the cause, whatever their outcomes. -// A run that recorded no incident has nothing to finish. +// Finish records action outcomes. It requests a root-cause fix if at least +// threshold incidents in the target's recent history share the current cause, +// regardless of their action outcomes. A run with no recorded incident is ignored. func (m *Module) Finish(ctx context.Context, runID uuid.UUID, actions []Action, threshold int) (Review, error) { current, ok, err := m.ledger.Incident(ctx, m.orgID, runID) if err != nil || !ok { @@ -194,8 +191,8 @@ func incident(r store.Incident) (Incident, error) { return out, nil } -// proposalBody asks for the cause to be fixed rather than handled again. same -// is newest first. +// proposalBody expects matching incidents in newest-first order so it can +// report the date of the oldest match. func proposalBody(current store.Incident, same []store.Incident) string { var tried []string for _, r := range same { @@ -209,10 +206,10 @@ func proposalBody(current store.Incident, same []store.Incident) string { } } } - body := fmt.Sprintf("%q has been the cause of %d incidents on %s %s since %s. Handling each one has not stopped it, so find what keeps producing it and fix that.", + body := fmt.Sprintf("%q caused %d recorded incidents on %s %s since %s. Identify and fix the underlying problem.", current.Cause, len(same), current.TargetKind, current.TargetID, same[len(same)-1].At.Format("2006-01-02")) if len(tried) > 0 { - body += " Handled so far with: " + strings.Join(tried, "; ") + "." + body += " Recorded actions: " + strings.Join(tried, "; ") + "." } return body } diff --git a/modules/memory/memory_test.go b/modules/memory/memory_test.go index 5e8e4656f..d0284c816 100644 --- a/modules/memory/memory_test.go +++ b/modules/memory/memory_test.go @@ -38,9 +38,8 @@ func (f *fakeLedger) Propose(_ context.Context, _ string, p store.Proposal) (sto return p, nil } -// TestFinishProposesOnRecurrence guards the rule that a cause seen threshold -// times on a target raises a root-cause fix, whatever happened to the actions -// taken in each incident, and that other causes don't count towards it. +// TestFinishProposesOnRecurrence checks that only incidents with the same +// cause count towards the configured threshold, even if an action was denied. func TestFinishProposesOnRecurrence(t *testing.T) { now := time.Date(2026, 11, 16, 0, 0, 0, 0, time.UTC) run := uuid.New() @@ -60,8 +59,8 @@ func TestFinishProposesOnRecurrence(t *testing.T) { {"one short of the threshold", incidents("disk-pressure", "disk-pressure", "disk-pressure", "disk-pressure"), 5, false}, {"the threshold is met by this incident", incidents("disk-pressure", "disk-pressure", "disk-pressure", "disk-pressure", "disk-pressure"), 5, true}, {"other causes are not counted", incidents("disk-pressure", "oom", "oom", "oom", "oom", "disk-pressure"), 5, false}, - {"the threshold is the organisation's", incidents("disk-pressure", "disk-pressure", "disk-pressure"), 3, true}, - {"past the threshold still proposes", incidents("disk-pressure", "disk-pressure", "disk-pressure", "disk-pressure", "disk-pressure", "disk-pressure"), 5, true}, + {"uses the organisation's threshold", incidents("disk-pressure", "disk-pressure", "disk-pressure"), 3, true}, + {"proposes above the threshold", incidents("disk-pressure", "disk-pressure", "disk-pressure", "disk-pressure", "disk-pressure", "disk-pressure"), 5, true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { diff --git a/modules/runs/agent.go b/modules/runs/agent.go index 15b203094..f82f28088 100644 --- a/modules/runs/agent.go +++ b/modules/runs/agent.go @@ -2,6 +2,7 @@ package runs import ( "context" + "encoding/json/jsontext" "errors" "github.com/samcharles93/ai-sdk/agentloop" @@ -22,6 +23,11 @@ type AgentRequest struct { Prompt string Tools core.ToolSet Budget agentloop.Budget + // Capture is the name of a tool whose input the loop should keep. The + // input of its last successful call comes back as AgentResult.Returned. + // It is read from the journal, so a resumed run that replays the call + // still has it. + Capture string } // AgentResult is how an agent loop ended. @@ -33,6 +39,9 @@ type AgentResult struct { Steps int `json:"steps"` ToolCalls int `json:"tool_calls"` Usage chat.Usage `json:"usage"` + // Returned is the input of the agent's last successful call to the tool + // that AgentRequest.Capture names. + Returned jsontext.Value `json:"returned,omitzero"` } // Agent runs a tool-using agent loop whose every model call and tool call is @@ -76,6 +85,30 @@ func (rc *Context) Agent(req AgentRequest) (AgentResult, error) { if err := rc.ctx.Err(); err != nil { return out, err } + if req.Capture != "" && out.Completed { + if out.Returned, err = returned(rc.ctx, journal, req.Capture); err != nil { + return out, err + } + } + return out, nil +} + +// returned only scans entries consumed by the loop. On replay, the next +// position may already contain the workflow step that follows the loop. +func returned(ctx context.Context, j *Journal[agentloop.Entry], tool string) (jsontext.Value, error) { + var out jsontext.Value + end := int64(j.rc.next - j.base) + for pos := range end { + e, err := j.Lookup(ctx, pos) + switch { + case err != nil: + return nil, err + case e == nil: + return out, nil + case e.Kind == agentloop.EntryTool && e.ToolName == tool && e.Error == "" && !e.Unknown: + out = jsontext.Value(e.Input) + } + } return out, nil } @@ -86,7 +119,7 @@ func retryModel(p chat.Provider) chat.Provider { return middleware.RetryChat(middleware.RetryConfig{MaxAttempts: 4}, middleware.ExponentialBackoff{}, middleware.DefaultRetryableError)(p) } -// loopJournal is the run journal in the shape the agent loop consults. +// loopJournal adapts the run journal for the agent loop. type loopJournal struct { *Journal[agentloop.Entry] } diff --git a/modules/runs/agent_result.go b/modules/runs/agent_result.go new file mode 100644 index 000000000..e22e37e1b --- /dev/null +++ b/modules/runs/agent_result.go @@ -0,0 +1,108 @@ +package runs + +import ( + "context" + "encoding/json/jsontext" + json "encoding/json/v2" + "errors" + "fmt" + "strings" + + "github.com/google/jsonschema-go/jsonschema" + "github.com/samcharles93/ai-sdk/core" + + "github.com/samcharles93/archie-core/modules/connectors" +) + +// resultTool is the name of the tool that accepts a structured result. +const resultTool = "return_result" + +const resultRule = "Return your result by calling " + resultTool + " exactly once, when you have it." + +// resultSchema compiles the step's result schema. Tool parameters must be +// a JSON object. +func (s agentSettings) resultSchema() ([]byte, *jsonschema.Resolved, error) { + // Tool schemas are part of the journaled model request digest. + raw, err := json.Marshal(s.Result, json.Deterministic(true)) + if err != nil { + return nil, nil, fmt.Errorf("schema: %w", err) + } + var schema jsonschema.Schema + if err := json.Unmarshal(raw, &schema); err != nil { + return nil, nil, fmt.Errorf("schema: %w", err) + } + if schema.Type != "object" { + return nil, nil, errors.New("a JSON Schema with type object") + } + resolved, err := schema.Resolve(nil) + if err != nil { + return nil, nil, fmt.Errorf("schema: %w", err) + } + return raw, resolved, nil +} + +// resultFields are the fields the result schema declares, or nil when it +// declares none, which accepts any field. +func (s agentSettings) resultFields() map[string]bool { + props, _ := s.Result["properties"].(map[string]any) + if len(props) == 0 { + return nil + } + fields := make(map[string]bool, len(props)) + for name := range props { + fields[name] = true + } + return fields +} + +// system is the part of the agent's system prompt the step adds: its mission, +// its extra rules and, with a result schema, how to return it. +func (s agentSettings) system() string { + parts := []string{s.Mission} + if strings.TrimSpace(s.ExtraRules) != "" { + parts = append(parts, s.ExtraRules) + } + if s.Result != nil { + parts = append(parts, resultRule) + } + return strings.Join(parts, "\n\n") +} + +// newResultTool returns validation errors to the agent so it can correct +// its result. +func newResultTool(raw []byte, schema *jsonschema.Resolved) *core.Tool { + return &core.Tool{ + Name: resultTool, + Description: "Return the result of this step.", + Parameters: raw, + Execute: func(_ context.Context, input string) (string, error) { + if _, err := decodeResult(jsontext.Value(input), schema); err != nil { + return "", err + } + return "recorded", nil + }, + } +} + +// decodeResult reads what the agent returned and checks it against the schema. +func decodeResult(raw jsontext.Value, schema *jsonschema.Resolved) (map[string]any, error) { + var out map[string]any + if err := json.Unmarshal(raw, &out); err != nil { + return nil, fmt.Errorf("the result is not a JSON object: %w", err) + } + if err := schema.Validate(out); err != nil { + return nil, fmt.Errorf("the result doesn't match its schema: %w", err) + } + return out, nil +} + +// readOnly filters tool declarations to read-only tools. +func readOnly(decls []connectors.Declaration) []connectors.Declaration { + out := make([]connectors.Declaration, 0, len(decls)) + for _, d := range decls { + if d.ReadOnly { + out = append(out, d) + } + } + return out +} diff --git a/modules/runs/agent_test.go b/modules/runs/agent_test.go index a9a7bd794..d67df31ec 100644 --- a/modules/runs/agent_test.go +++ b/modules/runs/agent_test.go @@ -6,6 +6,8 @@ import ( json "encoding/json/v2" "errors" "log/slog" + "os" + "slices" "strings" "sync" "testing" @@ -14,9 +16,98 @@ import ( "github.com/samcharles93/ai-sdk/chat" "github.com/samcharles93/ai-sdk/core" + "github.com/samcharles93/archie-core/modules/governance" "github.com/samcharles93/archie-core/modules/runs/internal/store" ) +type triageModel struct { + scriptedModel + models []string +} + +func (m *triageModel) Chat(_ context.Context, req chat.Request) (chat.Response, error) { + m.models = append(m.models, req.Model) + if req.Messages[len(req.Messages)-1].Role == chat.RoleTool { + return chat.Response{Role: chat.RoleAssistant, Content: "No change needed.", FinishReason: "stop"}, nil + } + return chat.Response{ + Role: chat.RoleAssistant, FinishReason: "tool_calls", + ToolCalls: []chat.ToolCall{{ID: "classification", Name: "return_result", Arguments: `{"needs_code_change":false,"reasons":"Already resolved"}`}}, + }, nil +} + +type modelSettings struct { + onePersonPath + model *triageModel + name string +} + +func (s modelSettings) AgentModelForAlias(_ context.Context, alias string) (governance.Model, error) { + if alias != "fast" { + return governance.Model{}, governance.ErrNoAgentModel + } + return governance.Model{Model: s.name}, nil +} + +func (s modelSettings) AgentProvider(context.Context, governance.Model) (chat.Provider, chat.ModelInfo, error) { + return s.model, chat.ModelInfo{}, nil +} + +func (modelSettings) AttemptsPerRun(context.Context) (int, error) { return 6, nil } + +// TestTriageClassificationUsesPinnedModel checks that a resumed step returns +// the same typed classification even if its model alias now resolves differently. +func TestTriageClassificationUsesPinnedModel(t *testing.T) { + raw, err := os.ReadFile("testdata/corpus/triage.yaml") + if err != nil { + t.Fatal(err) + } + body, _, ok := strings.Cut(string(raw), " - type: forge.comment") + if !ok { + t.Fatal("triage fixture has no forge comment step") + } + body = strings.Replace(body, " read_only: true", " read_only: true\n model: fast", 1) + doc, err := ParseDocument(body) + if err != nil { + t.Fatal(err) + } + engine := newEngine(nil, &fakeWork{}, slog.New(slog.DiscardHandler), Options{}) + engine.Use(Deps{Settings: modelSettings{}}) + _, err = engine.Install(t.Context(), "org", "triage", strings.Replace(body, "model: fast", "model: missing", 1), "operator") + var invalid *InvalidError + if !errors.As(err, &invalid) || len(invalid.Problems) != 1 || invalid.Problems[0].Path != "steps[0].settings.model" { + t.Fatalf("saving an unknown alias returned %v, want a model setting error", err) + } + run := store.Run{ID: uuid.New()} + work := &fakeWork{} + aliases, withMemory := agentUse(doc) + for _, tt := range []struct { + name string + want []string + }{ + {name: "fast-v1", want: []string{"fast-v1", "fast-v1"}}, + {name: "fast-v2"}, + } { + model := &triageModel{} + deps := Deps{Settings: modelSettings{model: model, name: tt.name}, Toolbox: anyBox{}, Judge: &actionJudge{}} + rc := newContext(t.Context(), work, run, uuid.New(), work.steps) + p := &progress{results: make(map[string]StepResult)} + if err := deps.pin(rc, p, aliases, withMemory); err != nil { + t.Fatal(err) + } + if err := deps.runStep(rc, p, doc.Steps[0], 0); err != nil { + t.Fatal(err) + } + got := p.results["classify"].Result + if got["needs_code_change"] != false || got["reasons"] != "Already resolved" { + t.Fatalf("classification = %#v", got) + } + if !slices.Equal(model.models, tt.want) { + t.Fatalf("model calls = %v, want %v", model.models, tt.want) + } + } +} + // scriptedModel answers by turn: a tool call first, then findings once the // tool's result is in the history. type scriptedModel struct { diff --git a/modules/runs/checks.go b/modules/runs/checks.go index 5fe2f7da7..b3e1d8fe2 100644 --- a/modules/runs/checks.go +++ b/modules/runs/checks.go @@ -42,12 +42,24 @@ func (c *checker) step(at string, s Step, mode RepositoryMode, earlier *refScope c.typed(at, s, mode, earlier) } if s.ID != "" { - earlier.steps[s.ID] = nil + earlier.steps[s.ID] = resultFields(s) } } -// controls validates the settings every step carries: when, on_failure and -// retry. +// resultFields are the result fields a step's type declares, or nil when it +// declares none. +func resultFields(s Step) map[string]bool { + if s.Type != typeAgent { + return nil + } + var settings agentSettings + if decodeSettings(s.Settings, &settings) != nil { + return nil + } + return settings.resultFields() +} + +// controls validates the common step settings: when, on_failure and retry. func (c *checker) controls(at string, s Step, earlier *refScope) { if s.OnFailure != "" && s.OnFailure != onFailurePark && s.OnFailure != onFailureContinue { c.add(at+".on_failure", "park or continue, not %q", s.OnFailure) diff --git a/modules/runs/corpus_test.go b/modules/runs/corpus_test.go index c987bbc9a..db04d25e9 100644 --- a/modules/runs/corpus_test.go +++ b/modules/runs/corpus_test.go @@ -38,14 +38,14 @@ func TestCorpusChecksAsItsStepTypesLand(t *testing.T) { }{ {name: "investigate checks", file: "investigate"}, { - name: "feasibility waits for repositories, switch, approve and agent results", file: "feasibility", - want: slices.Concat(notYet("repo.prepare", "workflow.approve"), agentLater("read_only", "result", "read_only", "result"), + name: "feasibility waits for repositories, switch and approve", file: "feasibility", + want: slices.Concat(notYet("repo.prepare", "workflow.approve"), []string{"switch steps are not supported yet"}), }, { - name: "implement waits for repositories, gates and agent settings", file: "implement", + name: "implement waits for repositories and gates", file: "implement", want: slices.Concat(notYet("repo.prepare", "repo.tooling", "gate.repository", "repo.commit", "repo.open-pr"), - agentLater("read_only", "gate", "extra_rules")), + agentLater("gate")), }, {name: "pr-review waits for review steps", file: "pr-review", want: notYet("review.pr")}, { @@ -53,13 +53,34 @@ func TestCorpusChecksAsItsStepTypesLand(t *testing.T) { want: slices.Concat(notYet("review.start-round", "repo.prepare", "repo.commit", "review.reply"), agentLater("gate")), }, { - name: "tdd waits for repositories, gates and agent settings", file: "tdd", + name: "tdd waits for repositories, gates and protected tests", file: "tdd", want: slices.Concat(notYet("repo.prepare", "repo.tooling", "gate.repository", "gate.repository", "repo.commit", "repo.commit", "repo.open-pr"), - agentLater("read_only", "gate", "gate", "protect_tests", "extra_rules")), + agentLater("gate", "gate", "protect_tests")), }, { name: "triage's references and conditions check; it waits for forge and outcome steps", file: "triage", - want: slices.Concat(notYet("forge.comment", "forge.close-issue", "workflow.finish", "workflow.handoff"), agentLater("read_only", "result")), + want: notYet("forge.comment", "forge.close-issue", "workflow.finish", "workflow.handoff"), + }, + { + name: "a result field the agent's schema doesn't declare is refused", + yaml: `steps: + - id: assess + type: agent.run + settings: + mission: m + result: {type: object, properties: {fit: {type: boolean}}} + - type: agent.run + settings: {mission: "{{ steps.assess.result.fit }} {{ steps.assess.result.verdict }}"} +`, + want: []string{`step "assess"'s result has no field "verdict"`}, + }, + { + name: "a result schema must describe an object", + yaml: `steps: + - type: agent.run + settings: {mission: m, result: {type: string}} +`, + want: []string{"result: a JSON Schema with type object"}, }, { name: "an earlier step's summary and result fields resolve", diff --git a/modules/runs/definition.go b/modules/runs/definition.go index 0f97b6026..b8977aa04 100644 --- a/modules/runs/definition.go +++ b/modules/runs/definition.go @@ -15,13 +15,12 @@ import ( //go:embed workflows/investigate.yaml var investigateDocument string -// InvestigateDocument is the workflow document that has an agent look into the -// event that started the run, put what it proposes to people, and report. +// InvestigateDocument defines the default workflow for investigating an event, +// requesting approval for proposed actions and reporting the findings. func InvestigateDocument() string { return investigateDocument } -// Document is a workflow as data: what it declares to whatever starts it, and -// the ordered steps the engine executes. It is stored, diffed and validated as -// text. +// Document defines a workflow's interface and ordered steps. The engine +// stores, compares and validates workflow definitions as text. type Document struct { Interface `yaml:",inline"` Steps []Step `yaml:"steps"` @@ -68,8 +67,8 @@ const ( onFailureContinue = "continue" ) -// name is what a step's work is recorded under: its id, else its type and -// position, which a pinned document keeps stable. +// name returns the journal name. Steps without an ID use their type and +// position, which stay stable because the run pins its workflow version. func (s Step) name(index int) string { if s.ID != "" { return s.ID @@ -77,7 +76,7 @@ func (s Step) name(index int) string { return fmt.Sprintf("%s@%d", s.Type, index) } -// Problem is one thing wrong with a document, at the path of the field. +// Problem describes a validation issue at a field path. type Problem struct { Path string `json:"path"` Message string `json:"message"` @@ -85,7 +84,7 @@ type Problem struct { func (p Problem) String() string { return p.Path + ": " + p.Message } -// InvalidError is the problems found in a document. +// InvalidError contains the problems found during workflow validation. type InvalidError struct { Problems []Problem `json:"problems"` } @@ -174,7 +173,7 @@ func decodeSettings(settings yaml.Node, target any) error { var unknownSetting = regexp.MustCompile(`field (\S+) not found`) -// Describe says in plain language what each step does. +// Describe returns a plain-language summary of each step. func (d Document) Describe() []string { out := make([]string, 0, len(d.Steps)) for _, s := range d.Steps { @@ -191,7 +190,8 @@ func (d Document) Describe() []string { return out } -// normalise is the form two documents are compared in. +// normalise ignores line-ending differences and surrounding whitespace when +// comparing workflow documents. func normalise(body string) string { return string(bytes.TrimSpace([]byte(strings.ReplaceAll(body, "\r\n", "\n")))) } diff --git a/modules/runs/dry.go b/modules/runs/dry.go index 564286031..aefa33982 100644 --- a/modules/runs/dry.go +++ b/modules/runs/dry.go @@ -17,8 +17,8 @@ import ( "github.com/samcharles93/archie-core/modules/runs/internal/store" ) -// dryOverlay is what a dry run carries instead of a stored workflow version: -// the document it runs, and the policy it is judged under as if published. +// dryOverlay contains the workflow document and policy a dry run tests as +// if they were published. type dryOverlay struct { Document string `json:"document"` PolicyKind string `json:"policy_kind,omitzero"` @@ -32,7 +32,6 @@ func (o *dryOverlay) policy() *judgement.PolicyOverlay { return &judgement.PolicyOverlay{Kind: o.PolicyKind, Body: o.Policy} } -// readOverlay decodes the overlay a dry run carries. func readOverlay(raw jsontext.Value) (*dryOverlay, error) { var o dryOverlay if err := json.Unmarshal(raw, &o); err != nil { @@ -41,12 +40,10 @@ func readOverlay(raw jsontext.Value) (*dryOverlay, error) { return &o, nil } -// Dry reports whether the run is a dry run: every action is judged and none -// that changes something is executed, nobody is asked and nothing is sent. +// Dry reports whether the run is a dry run. Dry runs evaluate actions but +// execute only read-only actions, and send no approval requests or notifications. func (rc *Context) Dry() bool { return rc.run.Dry } -// mayExecute reports whether judgement's decision may be carried out in this -// run. A dry run carries out only what reads. func (rc *Context) mayExecute(d judgement.Decision) bool { return !rc.run.Dry || d.Action.Effect.ReadOnly } @@ -76,7 +73,7 @@ type DryRequest struct { // StartDry queues a dry run of a document and returns its run's id. A document // that doesn't validate starts nothing. func (e *Engine) StartDry(ctx context.Context, req DryRequest) (uuid.UUID, error) { - if _, err := ParseDocument(req.Document); err != nil { + if err := e.validateDocument(ctx, req.Document); err != nil { return uuid.Nil, err } overlay, err := json.Marshal(dryOverlay{Document: req.Document, PolicyKind: req.PolicyKind, Policy: req.Policy}) @@ -124,7 +121,7 @@ type DryNotice struct { Body string `json:"body"` } -// DryReport is what a dry run did: every action it judged and what came of it. +// DryReport records evaluated actions, their outcomes and unsent notifications. type DryReport struct { RunID uuid.UUID `json:"run_id"` State string `json:"state"` @@ -135,8 +132,8 @@ type DryReport struct { At time.Time `json:"at"` } -// DryReport reads what dry run id did so far. ok is false when there is no -// such dry run. +// DryReport returns the results recorded so far. It reports ok=false if the +// ID does not identify a dry run. func (e *Engine) DryReport(ctx context.Context, orgID string, id uuid.UUID) (rep DryReport, ok bool, err error) { run, found, err := e.db.Detail(ctx, orgID, id) if err != nil || !found || !run.Dry { @@ -205,7 +202,7 @@ func pathNames(path []governance.PathStep) []string { return names } -// settleDry records who each held action would be put to, and asks nobody. +// settleDry records the intended approvers without sending requests. func settleDry(rc *Context, judge Judge, settings Settings, settled map[string]bool) ([]approvalOutcome, error) { held, err := heldSince(rc, judge, settled) if err != nil { diff --git a/modules/runs/escalation_test.go b/modules/runs/escalation_test.go index f0e85ea23..d4537559f 100644 --- a/modules/runs/escalation_test.go +++ b/modules/runs/escalation_test.go @@ -15,10 +15,8 @@ import ( "github.com/samcharles93/archie-core/modules/runs/internal/store" ) -// TestAfterApprovalDiscardsAnApprovalThatNoLongerMatchesTheAction guards the -// rule that an approval covers the action as it was shown: when something else -// changed the target or effect, or judgement got worse, the approval is -// discarded and the action is put to people again, never run on the old one. +// TestAfterApprovalDiscardsAnApprovalThatNoLongerMatchesTheAction checks that +// a changed target or effect, or a stricter verdict, invalidates the approval. func TestAfterApprovalDiscardsAnApprovalThatNoLongerMatchesTheAction(t *testing.T) { decided := func(level judgement.Level, mutate func(*judgement.Decision)) judgement.Decision { d := judgement.Decision{Level: level, Action: judgement.Action{ @@ -36,15 +34,15 @@ func TestAfterApprovalDiscardsAnApprovalThatNoLongerMatchesTheAction(t *testing. fresh judgement.Decision want afterwards }{ - {name: "an unchanged action runs on its approval", fresh: decided(judgement.ProposeAndWait, nil), want: runApproval}, - {name: "a worse verdict is put to people again", fresh: decided(judgement.Escalate, nil), want: reaskPerson}, - {name: "a changed target is put to people again", fresh: decided(judgement.ProposeAndWait, func(d *judgement.Decision) { + {name: "an unchanged action uses the existing approval", fresh: decided(judgement.ProposeAndWait, nil), want: runApproval}, + {name: "a stricter verdict requires new approval", fresh: decided(judgement.Escalate, nil), want: reaskPerson}, + {name: "a changed target requires new approval", fresh: decided(judgement.ProposeAndWait, func(d *judgement.Decision) { d.Action.Target.ID = "db" }), want: reaskPerson}, - {name: "a changed environment is put to people again", fresh: decided(judgement.ProposeAndWait, func(d *judgement.Decision) { + {name: "a changed environment requires new approval", fresh: decided(judgement.ProposeAndWait, func(d *judgement.Decision) { d.Action.Target.Environment = "staging" }), want: reaskPerson}, - {name: "a changed effect is put to people again", fresh: decided(judgement.ProposeAndWait, func(d *judgement.Decision) { + {name: "a changed effect requires new approval", fresh: decided(judgement.ProposeAndWait, func(d *judgement.Decision) { d.Action.Effect.Summary = "delete" }), want: reaskPerson}, {name: "a refusal discards the approval", fresh: decided(judgement.Never, nil), want: refuseAction}, @@ -118,7 +116,7 @@ func TestWaitThroughPathMovesOnAtEachDeadlineAndExpiresAtTheEnd(t *testing.T) { {name: "one step with no answer expires without escalating", path: []governance.PathStep{step("a")}, wantExpired: 1}, {name: "every step runs out and then the request expires", path: []governance.PathStep{step("a"), step("b"), step("c")}, wantSteps: []int{1, 2}, wantExpired: 1}, {name: "an answer at the second step stops the escalation", path: []governance.PathStep{step("a"), step("b"), step("c")}, arrives: 2, wantSteps: []int{1}, wantAnswered: true}, - {name: "an answer at the first step escalates nowhere", path: []governance.PathStep{step("a"), step("b")}, arrives: 1, wantAnswered: true}, + {name: "an answer at the first step prevents escalation", path: []governance.PathStep{step("a"), step("b")}, arrives: 1, wantAnswered: true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { diff --git a/modules/runs/internal/store/store.go b/modules/runs/internal/store/store.go index 808ec53c1..7d5477ae4 100644 --- a/modules/runs/internal/store/store.go +++ b/modules/runs/internal/store/store.go @@ -50,8 +50,9 @@ type Run struct { Attempts int `db:"attempts"` // Dry marks a dry run, and Overlay is the document and policy it runs // under; a dry run pins no stored version. - Dry bool `db:"dry"` - Overlay jsontext.Value `db:"overlay"` + Dry bool `db:"dry"` + // Scan into []byte because live runs have a SQL NULL overlay. + Overlay []byte `db:"overlay"` } // Step is one recorded step of a run. diff --git a/modules/runs/interpreter.go b/modules/runs/interpreter.go index 073a995f4..8cc455007 100644 --- a/modules/runs/interpreter.go +++ b/modules/runs/interpreter.go @@ -12,6 +12,7 @@ import ( "strings" "time" + "github.com/google/jsonschema-go/jsonschema" "github.com/google/uuid" "github.com/samcharles93/ai-sdk/agentloop" "github.com/samcharles93/ai-sdk/chat" @@ -27,10 +28,9 @@ import ( "github.com/samcharles93/archie-core/modules/runs/internal/store" ) -// Settings is where a workflow finds what the organisation decided: the agent -// model it runs, and the escalation path an approval request takes. +// Settings provides the organisation's model configuration and escalation policy. type Settings interface { - AgentModel(ctx context.Context) (governance.Model, error) + AgentModelForAlias(ctx context.Context, alias string) (governance.Model, error) // AgentProvider builds the chat client for a model, resolving the // credential it names. The model is journaled; the client is rebuilt. AgentProvider(ctx context.Context, model governance.Model) (chat.Provider, chat.ModelInfo, error) @@ -55,10 +55,10 @@ type Judge interface { ForRun(ctx context.Context, runID uuid.UUID) ([]judgement.Decision, error) } -// Approver asks people to allow an action judgement held, moves a request -// that nobody answered to the next step of its path, and settles one whose last -// step ran out. An approval is counted when the action is about to run on it, -// and discarded when the action no longer matches it. +// Approver requests approval for actions held by judgement, escalates +// unanswered requests and expires them after the final deadline. An approval +// counts towards an increase in autonomy only when the action is about to run +// using it. If reassessment invalidates the approval, it is discarded instead. type Approver interface { Ask(ctx context.Context, runID uuid.UUID, key string, d judgement.Decision, path []governance.PathStep) (approvals.Request, error) Escalate(ctx context.Context, id uuid.UUID, step int, to governance.PathStep) error @@ -106,14 +106,18 @@ type Deps struct { // and the results of earlier steps. type Workflow func(rc *Context, input jsontext.Value) (jsontext.Value, error) -// pinned is what a run's first steps fix, so a resumed run asks the same model -// with the same tools even if settings or connectors changed meanwhile. +type pinnedModel struct { + model governance.Model + provider chat.Provider + info chat.ModelInfo +} + +// A resumed run reuses the recorded configuration but rebuilds model +// clients and resolves tool calls against the connectors currently running. type pinned struct { - model governance.Model + models map[string]pinnedModel threshold int decls []connectors.Declaration - provider chat.Provider - info chat.ModelInfo } // progress is what the steps of one execution have produced so far. @@ -141,15 +145,15 @@ type progress struct { // Compile returns the code that runs doc. The same steps run live and dry; a // dry run (rc.Dry) judges every action and executes only those that read. func (d Deps) Compile(doc Document) Workflow { - agent, withMemory := agentUse(doc) + aliases, withMemory := agentUse(doc) settleAtEnd := heldAfterLastApproval(doc) return func(rc *Context, input jsontext.Value) (jsontext.Value, error) { p := &progress{ input: input, inputs: decodeInputs(input), instructions: doc.Instructions, results: map[string]StepResult{}, outcomes: []approvalOutcome{}, settled: map[string]bool{}, } - if agent { - if err := d.pin(rc, p, withMemory); err != nil { + if len(aliases) > 0 { + if err := d.pin(rc, p, aliases, withMemory); err != nil { return nil, err } } @@ -173,9 +177,8 @@ func (d Deps) Compile(doc Document) Workflow { } } -// runStep runs one step if its condition holds, with its settings rendered -// from the run's inputs and earlier results, and keeps what it leaves under its -// id. +// runStep resolves settings from the run's inputs and earlier results, and +// saves the result under the step ID if the step runs. func (d Deps) runStep(rc *Context, p *progress, s Step, index int) error { name := s.name(index) values := scope(p.inputs, p.results) @@ -214,19 +217,28 @@ func stepFailed(err error) bool { return true } -// agentUse reports whether any step runs an agent, and whether one uses memory. -func agentUse(doc Document) (agent, withMemory bool) { +func agentUse(doc Document) (aliases []string, withMemory bool) { for _, s := range doc.Steps { if s.Type != typeAgent { continue } - agent = true var settings agentSettings - if decodeSettings(s.Settings, &settings) == nil && settings.Memory { - withMemory = true + if decodeSettings(s.Settings, &settings) == nil { + alias := modelAlias(settings.Model) + if !slices.Contains(aliases, alias) { + aliases = append(aliases, alias) + } + withMemory = withMemory || settings.Memory } } - return agent, withMemory + return aliases, withMemory +} + +func modelAlias(alias string) string { + if alias == "default" { + return "" + } + return alias } // heldAfterLastApproval reports whether a step that can propose an action comes @@ -254,19 +266,29 @@ func decodeInputs(input jsontext.Value) map[string]any { return out } -func (d Deps) pin(rc *Context, p *progress, withMemory bool) error { - model, err := rc.Step("agent-model", func(ctx context.Context, _ string) (governance.Model, error) { - return d.Settings.AgentModel(ctx) - }) - if err != nil { - return err - } - if withMemory { - if p.pin.threshold, err = rc.Step("recurrence-threshold", func(ctx context.Context, _ string) (int, error) { - return d.Settings.RecurrenceThreshold(ctx) - }); err != nil { +func (d Deps) pin(rc *Context, p *progress, aliases []string, withMemory bool) error { + models := make(map[string]pinnedModel, len(aliases)) + for _, alias := range aliases { + step := "agent-model" + if alias != "" { + step += ":" + alias + } + model, err := rc.Step(step, func(ctx context.Context, _ string) (governance.Model, error) { + return d.Settings.AgentModelForAlias(ctx, alias) + }) + if err != nil { return err } + models[alias] = pinnedModel{model: model} + } + if withMemory { + threshold, err := rc.Step("recurrence-threshold", func(ctx context.Context, _ string) (int, error) { + return d.Settings.RecurrenceThreshold(ctx) + }) + if err != nil { + return err + } + p.pin.threshold = threshold } decls, err := rc.Step("agent-tools", func(context.Context, string) ([]connectors.Declaration, error) { return d.Toolbox.Declarations(), nil @@ -274,11 +296,14 @@ func (d Deps) pin(rc *Context, p *progress, withMemory bool) error { if err != nil { return err } - provider, info, err := d.Settings.AgentProvider(rc.Context(), model) - if err != nil { - return err + for _, alias := range aliases { + m := models[alias] + if m.provider, m.info, err = d.Settings.AgentProvider(rc.Context(), m.model); err != nil { + return err + } + models[alias] = m } - p.pin.model, p.pin.decls, p.pin.provider, p.pin.info = model, decls, provider, info + p.pin.models, p.pin.decls = models, decls return nil } @@ -292,34 +317,54 @@ func agentSystem(instructions, mission string) string { return strings.Join(append(parts, mission), "\n\n") } -// runAgent has an agent work on its mission with the run's inputs. An action -// it proposes is judged as it calls it; the ones judgement holds wait for an -// approval step. +// runAgent evaluates each action when the agent calls it. Actions that +// require human approval are deferred to an approval step. func runAgent(d Deps, rc *Context, p *progress, name string, s agentSettings) (StepResult, error) { budget, err := stepBudget(rc.Context(), d.Settings) if err != nil { return StepResult{}, err } + decls := p.pin.decls + if s.ReadOnly { + decls = readOnly(decls) + } + var ( + resultRaw []byte + resultSchema *jsonschema.Resolved + ) + if s.Result != nil { + if resultRaw, resultSchema, err = s.resultSchema(); err != nil { + return StepResult{}, fmt.Errorf("step %q result: %w", name, err) + } + } tools := func() core.ToolSet { - set := toolSet(d.Toolbox, d.Judge, rc, p.pin.decls) + set := toolSet(d.Toolbox, d.Judge, rc, decls) if s.Memory { set = withMemory(set, d.Memory, rc.RunID()) } + if resultSchema != nil { + set[resultTool] = newResultTool(resultRaw, resultSchema) + } return set } + model := p.pin.models[modelAlias(s.Model)] base := AgentRequest{ - Provider: p.pin.provider, Model: p.pin.model.Model, ModelInfo: p.pin.info, - System: agentSystem(p.instructions, s.Mission), Tools: tools(), + Provider: model.provider, Model: model.model.Model, ModelInfo: model.info, + System: agentSystem(p.instructions, s.system()), Tools: tools(), } first := base first.Name, first.Prompt, first.Budget = name, string(p.input), budget + if resultSchema != nil { + first.Capture = resultTool + } res, err := rc.Agent(first) if err != nil { return StepResult{}, err } + returnedResult := res.Returned followUp := base followUp.Tools = tools() - if res, err = followThrough(rc, d.Judge, p.pin.decls, p.input, res, followUp); err != nil { + if res, err = followThrough(rc, d.Judge, decls, p.input, res, followUp); err != nil { return StepResult{}, err } p.agent = &res @@ -335,7 +380,23 @@ func runAgent(d Deps, rc *Context, p *progress, name string, s agentSettings) (S } } } - return StepResult{Summary: res.Answer}, nil + return agentResult(name, res, returnedResult, resultSchema) +} + +func agentResult(name string, res AgentResult, returned jsontext.Value, schema *jsonschema.Resolved) (StepResult, error) { + out := StepResult{Summary: res.Answer} + if schema == nil || !res.Completed { + return out, nil + } + if len(returned) == 0 { + return StepResult{}, fmt.Errorf("step %q: the agent finished without returning its result", name) + } + result, err := decodeResult(returned, schema) + if err != nil { + return StepResult{}, fmt.Errorf("step %q: %w", name, err) + } + out.Result = result + return out, nil } // result is the run's output: the last agent's findings, or what became of the @@ -464,9 +525,8 @@ func settleProposals(rc *Context, judge Judge, ask Approver, settings Settings, return out, nil } -// heldSince is what judgement held in the run that no earlier approval step -// settled. The whole list is recorded and the settled ones dropped after, so a -// replay filters the recorded list the same way. +// heldSince journals all actions held for approval before filtering out +// settled ones so that replay uses the same list. func heldSince(rc *Context, judge Judge, settled map[string]bool) ([]judgement.Decision, error) { held, err := rc.Step("proposals", func(ctx context.Context, _ string) ([]judgement.Decision, error) { decisions, err := judge.ForRun(ctx, rc.RunID()) @@ -481,8 +541,9 @@ func heldSince(rc *Context, judge Judge, settled map[string]bool) ([]judgement.D return slices.DeleteFunc(held, func(d judgement.Decision) bool { return settled[d.Action.Digest] }), nil } -// maxReapprovals bounds how many times an action that keeps changing under its -// approval is put to people again. +// maxReapprovals limits the number of new requests for one action after +// reassessment invalidates an approval and the action still requires approval. +// The initial request and escalation within a request do not count. const maxReapprovals = 2 // settleOne gets one action approved and runs it. Before it runs it is judged @@ -656,11 +717,9 @@ const ( refuseAction ) -// afterApproval says what an approval of approved means now that the action was -// judged again as fresh. An approval covers the action as it was shown, so one -// whose verdict got worse, or whose target or effect changed, no longer -// stands. The target and effect are what the connector resolves the action to -// now, which is what judgement has to tell that something else changed them. +// afterApproval checks whether an approval is still valid after reassessment. +// If the action still requires approval, a stricter verdict or a change to the +// resolved target or effect requires a new request. func afterApproval(approved, fresh judgement.Decision) (afterwards, string) { switch { case fresh.Level == judgement.Never: @@ -672,7 +731,7 @@ func afterApproval(approved, fresh judgement.Decision) (afterwards, string) { case fresh.Action.Effect != approved.Action.Effect: return reaskPerson, "the effect changed since it was approved" case fresh.Level > approved.Level: - return reaskPerson, "judgement got worse since it was approved: now " + fresh.Level.String() + return reaskPerson, "judgement is stricter than when the action was approved: now " + fresh.Level.String() } return runApproval, "" } diff --git a/modules/runs/memory.go b/modules/runs/memory.go index 8d73da4a2..86e4790fb 100644 --- a/modules/runs/memory.go +++ b/modules/runs/memory.go @@ -17,28 +17,26 @@ import ( "github.com/samcharles93/archie-core/modules/memory" ) -// Memory is what an investigation remembers and recalls about its target. +// Memory stores and retrieves incident history for an investigation target. type Memory interface { Past(ctx context.Context, runID uuid.UUID, t memory.Target) ([]memory.Incident, error) Record(ctx context.Context, runID uuid.UUID, event jsontext.Value, t memory.Target, cause, summary string) error Finish(ctx context.Context, runID uuid.UUID, actions []memory.Action, threshold int) (memory.Review, error) } -// recordCauseBudget bounds the turn that files the cause: one tool call and a -// reply. var recordCauseBudget = agentloop.Budget{MaxSteps: 4, MaxTokens: 30_000, WallClock: 3 * time.Minute} // maxRecalled is how many earlier incidents the agent is shown. const maxRecalled = 10 const targetSchema = `{"type":"object","properties":{ -"target_kind":{"type":"string","description":"what the incident is about, such as host"}, -"target_id":{"type":"string","description":"which one, such as web-1"}}, +"target_kind":{"type":"string","description":"the resource type, such as host"}, +"target_id":{"type":"string","description":"the resource ID, such as web-1"}}, "required":["target_kind","target_id"]}` const causeSchema = `{"type":"object","properties":{ -"target_kind":{"type":"string","description":"what the incident happened to, such as host"}, -"target_id":{"type":"string","description":"which one, such as web-1"}, +"target_kind":{"type":"string","description":"the resource type, such as host"}, +"target_id":{"type":"string","description":"the resource ID, such as web-1"}, "cause":{"type":"string","description":"two to four lowercase words joined by hyphens, such as disk-pressure"}, "summary":{"type":"string","description":"one sentence: what happened and why"}}, "required":["target_kind","target_id","cause","summary"]}` @@ -54,11 +52,10 @@ type causeInput struct { Summary string `json:"summary"` } -// pastIncidentsTool lets the agent ask what happened to a target before. func pastIncidentsTool(mem Memory, runID uuid.UUID) *core.Tool { return &core.Tool{ Name: "memory_past_incidents", - Description: "Lists earlier incidents on a target: when, the cause found, what was done and how it ended.", + Description: "Lists previous incidents for a target, including dates, causes, actions and outcomes.", Parameters: []byte(targetSchema), Execute: func(ctx context.Context, input string) (string, error) { var in targetInput @@ -82,10 +79,9 @@ func pastIncidentsTool(mem Memory, runID uuid.UUID) *core.Tool { } } -// recordCause has the agent name the target and cause of the investigation it -// just finished, in a turn of its own: a small model that is asked for four -// plain fields right after writing its findings fills them reliably, where one -// asked to remember a tool call at the end of a long investigation does not. +// recordCause uses a separate turn because small models are more reliable +// when asked for four fields after reporting than when asked to remember +// a tool call throughout the investigation. func recordCause(rc *Context, mem Memory, input jsontext.Value, res AgentResult, req AgentRequest) error { if !res.Completed { return nil @@ -97,7 +93,7 @@ func recordCause(rc *Context, mem Memory, input jsontext.Value, res AgentResult, req.Name, req.System, req.Prompt, req.Budget = "record-cause", prompts.RecordCause(), string(prompt), recordCauseBudget req.Tools = core.ToolSet{"record_cause": { Name: "record_cause", - Description: "Files the target and cause of this incident in memory.", + Description: "Records the incident target, cause and summary.", Parameters: []byte(causeSchema), Execute: func(ctx context.Context, raw string) (string, error) { var in causeInput @@ -110,8 +106,7 @@ func recordCause(rc *Context, mem Memory, input jsontext.Value, res AgentResult, return "recorded", nil }, }} - // A turn that doesn't file anything leaves the run without an incident; the - // investigation's findings stand. + // If the agent records no incident, keep the investigation findings. _, err = rc.Agent(req) return err } @@ -158,8 +153,8 @@ func firstText(parts ...string) string { return "" } -// remember files what the run did and checks whether its cause has recurred -// enough to ask for a root-cause fix. +// remember records the incident and checks whether it meets the recurrence +// threshold for proposing a root-cause fix. func remember(rc *Context, mem Memory, judge Judge, outcomes []approvalOutcome, threshold int) (memory.Review, error) { return rc.Step("remember", func(ctx context.Context, _ string) (memory.Review, error) { decisions, err := judge.ForRun(ctx, rc.RunID()) diff --git a/modules/runs/references.go b/modules/runs/references.go index 77c6ce9c4..c121ab2c5 100644 --- a/modules/runs/references.go +++ b/modules/runs/references.go @@ -11,7 +11,7 @@ import ( "github.com/samcharles93/archie-core/internal/platform/stableid" ) -// StepResult is what a finished step leaves for later steps to reference as +// StepResult is the output later steps can reference as // steps..summary and steps..result.. type StepResult struct { Summary string `json:"summary"` @@ -21,8 +21,8 @@ type StepResult struct { // referencePattern matches {{ path }} in a step's string settings. var referencePattern = regexp.MustCompile(`\{\{\s*([^{}\s]+)\s*\}\}`) -// soleReferencePattern matches a value that is exactly one reference, which is -// what lets it carry the referenced value's own type instead of its text. +// soleReferencePattern matches a standalone reference, which preserves the +// referenced value's type. var soleReferencePattern = regexp.MustCompile(`^\s*\{\{\s*([^{}\s]+)\s*\}\}\s*$`) // taskFields are the fields a task run fills in. They are accepted in any @@ -135,11 +135,9 @@ func scope(inputs map[string]any, results map[string]StepResult) map[string]any // renderSettings returns a copy of settings with every reference replaced by // its value in this run. // -// A value that is exactly one reference takes that reference's own type, so a -// boolean, number, array or object crosses from one step to the next intact; a -// reference embedded in prose is interpolated as text, the only thing a string -// can hold. A reference to an absent value renders empty, which is not an -// error: a skipped earlier step leaves nothing behind. +// A standalone reference preserves the value's type. References embedded in +// strings are interpolated as text. Missing values render empty because a +// skipped step produces no output. func renderSettings(settings yaml.Node, values map[string]any) yaml.Node { rendered := deepCopyNode(&settings) renderNode(rendered, values) @@ -147,8 +145,7 @@ func renderSettings(settings yaml.Node, values map[string]any) yaml.Node { } // renderNode replaces references in a settings node's string values, in place. -// Mapping keys are interpolated as text: a key names a field, so it has no -// typed value to carry. +// Mapping keys are interpolated as text even for standalone references. func renderNode(node *yaml.Node, values map[string]any) { switch node.Kind { case yaml.DocumentNode, yaml.SequenceNode: diff --git a/modules/runs/runs.go b/modules/runs/runs.go index 89c5606e9..d7642323a 100644 --- a/modules/runs/runs.go +++ b/modules/runs/runs.go @@ -17,10 +17,11 @@ import ( "github.com/google/uuid" "github.com/jmoiron/sqlx" + "github.com/samcharles93/archie-core/modules/governance" "github.com/samcharles93/archie-core/modules/runs/internal/store" ) -// StartRequest names the workflow to run and the organisation it runs for. +// StartRequest specifies the workflow and organisation for a new run. // A non-empty Key makes the start idempotent within the organisation. type StartRequest struct { OrgID string @@ -107,7 +108,7 @@ func (e *Engine) steps() (Deps, error) { // what the latest version already is, and returns the version in force. A body // that doesn't validate is stored as nothing. func (e *Engine) Install(ctx context.Context, orgID, name, body, by string) (int, error) { - if _, err := ParseDocument(body); err != nil { + if err := e.validateDocument(ctx, body); err != nil { return 0, err } latest, err := e.db.Workflow(ctx, orgID, name, 0) @@ -124,6 +125,40 @@ func (e *Engine) Install(ctx context.Context, orgID, name, body, by string) (int return w.Version, nil } +func (e *Engine) validateDocument(ctx context.Context, body string) error { + doc, err := ParseDocument(body) + if err != nil { + return err + } + var problems []Problem + for i, step := range doc.Steps { + if step.Type != typeAgent { + continue + } + var s agentSettings + if err := decodeSettings(step.Settings, &s); err != nil { + return err + } + if modelAlias(s.Model) == "" { + continue + } + deps, err := e.steps() + if err != nil { + return err + } + if _, err := deps.Settings.AgentModelForAlias(ctx, s.Model); err != nil { + if !errors.Is(err, governance.ErrNoAgentModel) { + return err + } + problems = append(problems, Problem{Path: fmt.Sprintf("steps[%d].settings.model", i), Message: fmt.Sprintf("unknown agent model alias %q", s.Model)}) + } + } + if len(problems) > 0 { + return &InvalidError{Problems: problems} + } + return nil +} + // Has reports whether a workflow is stored under name. func (e *Engine) Has(ctx context.Context, orgID, name string) (bool, error) { _, err := e.db.Workflow(ctx, orgID, name, 0) diff --git a/modules/runs/steps_test.go b/modules/runs/steps_test.go index f28a93fdf..a73dcce24 100644 --- a/modules/runs/steps_test.go +++ b/modules/runs/steps_test.go @@ -72,7 +72,7 @@ func (a *askCounter) Discard(context.Context, uuid.UUID, string) error { return // onePersonPath sends every request to one person. type onePersonPath struct{} -func (onePersonPath) AgentModel(context.Context) (governance.Model, error) { +func (onePersonPath) AgentModelForAlias(context.Context, string) (governance.Model, error) { return governance.Model{}, errors.New("no agent in these workflows") } diff --git a/modules/runs/testdata/corpus/investigate.yaml b/modules/runs/testdata/corpus/investigate.yaml index 98a4d0e89..f78b55ffd 100644 --- a/modules/runs/testdata/corpus/investigate.yaml +++ b/modules/runs/testdata/corpus/investigate.yaml @@ -13,12 +13,11 @@ steps: infrastructure. An investigation ends when the cause is found, not when the first figure is. - Archie remembers earlier incidents. Once you know which target the event - is about (a kind and an id, such as host web-1), call - memory_past_incidents with it before you dig further. If it lists - earlier incidents with the same cause, say so in your findings: cite - each by its date, what was done then and how it ended, and weigh that - before recommending the same action again. + Once you identify the target (a kind and ID, such as host web-1), pass + it to memory_past_incidents before continuing. In your findings, list any + previous incidents with the same cause, along with their dates, + actions and outcomes. Consider those outcomes before recommending + the same action again. The event is a claim, not a fact. Check it against what you measure, and if they disagree (an alert says a disk is full and it is at 40%), say so @@ -45,8 +44,9 @@ steps: - The likely cause, and how sure you are. - The evidence: what you looked at and what it showed. - What should happen next, naming the action and its target precisely. - Say what judgement answered if you called it. Don't claim a call you - didn't make: only a tool call counts, and a failed one says so. + If a tool call returns a judgement decision, report the decision and + any reasons it gives. Report failed tool calls as failures, and don't + claim to have called a tool you didn't use. If you couldn't find out, say what you tried and what is missing. - id: approve diff --git a/modules/runs/vocabulary.go b/modules/runs/vocabulary.go index 0446418f8..2c25190ad 100644 --- a/modules/runs/vocabulary.go +++ b/modules/runs/vocabulary.go @@ -12,6 +12,7 @@ import ( "gopkg.in/yaml.v3" "github.com/samcharles93/archie-core/modules/deliveries" + "github.com/samcharles93/archie-core/modules/governance" "github.com/samcharles93/archie-core/modules/judgement" ) @@ -24,14 +25,12 @@ const ( typeSleep = "workflow.sleep" ) -// StepType is one word of the workflow vocabulary: the type a step names, the -// settings it takes, and the code that runs it. +// StepType defines a workflow step's settings and implementation. type StepType struct { Name string `json:"name"` - // Title is the human name editors show for the step type. + // Title is the label shown in workflow editors. Title string `json:"title"` - // NeedsRepository is a type that runs only with a worktree, so a workflow - // whose repository isn't required can't use it. + // NeedsRepository restricts the step type to workflows that require a repository. NeedsRepository bool `json:"needs_repository"` // Settings is the JSON Schema of the type's settings. Settings *jsonschema.Schema `json:"settings"` @@ -84,7 +83,7 @@ var stepTypes = vocabulary( define(typeAgent, "Run an agent", agentSettings.valid, agentSettings.describe, runAgent), define(typeAction, "Run a connector action", actionSettings.valid, actionSettings.describe, runAction), define(typeApproval, "Wait for approval", noSettings.valid, func(noSettings) string { - return "Waits for approval before anything judgement holds runs" + return "Requests approval for actions held by judgement and runs approved actions" }, runApprovalWait), define(typeNotify, "Report the findings", notifySettings.valid, notifySettings.describe, runNotify), define(typeSleep, "Wait", sleepSettings.valid, sleepSettings.describe, runSleep), @@ -107,31 +106,46 @@ func StepTypes() []StepType { return out } -// agentSettings run a tool-using agent. Mission is what it is asked to do; -// Memory gives it the lookup of a target's earlier incidents and files what -// the run found and did once it ends. type agentSettings struct { - Mission string `yaml:"mission" json:"mission" jsonschema:"what the agent is asked to do"` - Memory bool `yaml:"memory,omitempty" json:"memory,omitempty" jsonschema:"recall a target's earlier incidents and file what this run found"` + Mission string `yaml:"mission" json:"mission" jsonschema:"the task the agent should perform"` + Memory bool `yaml:"memory,omitempty" json:"memory,omitempty" jsonschema:"retrieve previous incidents for the target and store new findings"` + ReadOnly bool `yaml:"read_only,omitempty" json:"read_only,omitempty" jsonschema:"restrict the agent to read-only tools"` + ExtraRules string `yaml:"extra_rules,omitempty" json:"extra_rules,omitempty" jsonschema:"appended to the agent's system prompt"` + Model string `yaml:"model,omitempty" json:"model,omitempty" jsonschema:"organisation model alias; omit to use the default"` + // Result defines the output schema. Later steps can read the validated + // output at steps..result. + Result map[string]any `yaml:"result,omitempty" json:"result,omitempty" jsonschema:"JSON Schema for the object the agent must return; later steps read the result at steps..result"` } func (s agentSettings) valid() error { if s.Mission == "" { return errors.New("an agent needs a mission") } + if err := governance.ValidateAgentModelAlias(s.Model); err != nil { + return err + } + if s.Result != nil { + if _, _, err := s.resultSchema(); err != nil { + return fmt.Errorf("result: %w", err) + } + } return nil } func (s agentSettings) describe() string { - line := "An agent works on its mission, looking things up and proposing actions, each judged before it runs" + line := "The agent looks up information and proposes actions for judgement before execution" + if s.ReadOnly { + line = "The agent can only use read-only tools" + } + if s.Result != nil { + line += "; it returns a structured result" + } if s.Memory { - line += "; it recalls earlier incidents and files what it found" + line += "; it retrieves previous incidents and stores new findings" } return line } -// actionSettings judge one connector action and run it if judgement lets it -// run unattended. type actionSettings struct { Connector string `yaml:"connector" json:"connector"` Name string `yaml:"name" json:"name"` @@ -146,21 +160,19 @@ func (s actionSettings) valid() error { } func (s actionSettings) describe() string { - return fmt.Sprintf("Runs %s %s once judged", s.Connector, s.Name) + return fmt.Sprintf("Submits %s %s for judgement and runs it if allowed to run unattended", s.Connector, s.Name) } -// noSettings is a type that takes none. type noSettings struct{} func (noSettings) valid() error { return nil } -// notifySettings tell people what the run found. Title replaces the default. type notifySettings struct { Title string `yaml:"title,omitempty" json:"title,omitempty"` } func (notifySettings) valid() error { return nil } -func (notifySettings) describe() string { return "Reports what it found and did" } +func (notifySettings) describe() string { return "Reports the run's findings and actions" } // maxSleep bounds a sleep step. const maxSleep = 24 * time.Hour @@ -184,9 +196,8 @@ func (s sleepSettings) duration() time.Duration { return d } -// runAction judges the action and runs it if judgement lets it run -// unattended. One judgement holds waits for an approval step; one it refuses -// is reported. +// runAction executes only actions allowed to run unattended. A later +// approval step handles actions held for approval. func runAction(d Deps, rc *Context, p *progress, name string, s actionSettings) (StepResult, error) { input := s.Input if input == nil { @@ -220,7 +231,6 @@ func runAction(d Deps, rc *Context, p *progress, name string, s actionSettings) }, nil } -// runApprovalWait puts what judgement held to people and runs what they allow. func runApprovalWait(d Deps, rc *Context, p *progress, _ string, _ noSettings) (StepResult, error) { out, err := d.settle(rc, p) if err != nil { @@ -229,9 +239,8 @@ func runApprovalWait(d Deps, rc *Context, p *progress, _ string, _ noSettings) ( return StepResult{Summary: outcomeLines(out)}, nil } -// settle puts every action judgement held and nobody has settled yet to -// people, and runs what they allow. In a dry run nobody is asked: it records -// who would be. +// settle requests approval for pending actions and runs those approved. +// A dry run records the intended recipients without contacting them. func (d Deps) settle(rc *Context, p *progress) ([]approvalOutcome, error) { var ( out []approvalOutcome @@ -252,8 +261,9 @@ func (d Deps) settle(rc *Context, p *progress) ([]approvalOutcome, error) { return out, nil } -// runNotify tells people what the run found, after filing it in memory. In a -// dry run it records what would be sent and sends nothing. +// runNotify records incident history before notifying when memory is enabled +// and the investigation can finish. A dry run records the notification without +// sending it. func runNotify(d Deps, rc *Context, p *progress, _ string, s notifySettings) (StepResult, error) { if p.memory && !p.remembered && !p.stopped && !rc.Dry() { review, err := remember(rc, d.Memory, d.Judge, p.outcomes, p.pin.threshold) @@ -274,7 +284,7 @@ func runNotify(d Deps, rc *Context, p *progress, _ string, s notifySettings) (St _, err := rc.Step("notify", func(ctx context.Context, key string) (bool, error) { return true, d.Notifier.Notify(ctx, n(key)) }) - // The title doesn't depend on the key, so a replay summarises it alike. + // The title doesn't depend on the key, so it stays the same on replay. return StepResult{Summary: n("").Title}, err } diff --git a/modules/runs/workflows/investigate.yaml b/modules/runs/workflows/investigate.yaml index 98a4d0e89..f78b55ffd 100644 --- a/modules/runs/workflows/investigate.yaml +++ b/modules/runs/workflows/investigate.yaml @@ -13,12 +13,11 @@ steps: infrastructure. An investigation ends when the cause is found, not when the first figure is. - Archie remembers earlier incidents. Once you know which target the event - is about (a kind and an id, such as host web-1), call - memory_past_incidents with it before you dig further. If it lists - earlier incidents with the same cause, say so in your findings: cite - each by its date, what was done then and how it ended, and weigh that - before recommending the same action again. + Once you identify the target (a kind and ID, such as host web-1), pass + it to memory_past_incidents before continuing. In your findings, list any + previous incidents with the same cause, along with their dates, + actions and outcomes. Consider those outcomes before recommending + the same action again. The event is a claim, not a fact. Check it against what you measure, and if they disagree (an alert says a disk is full and it is at 40%), say so @@ -45,8 +44,9 @@ steps: - The likely cause, and how sure you are. - The evidence: what you looked at and what it showed. - What should happen next, naming the action and its target precisely. - Say what judgement answered if you called it. Don't claim a call you - didn't make: only a tool call counts, and a failed one says so. + If a tool call returns a judgement decision, report the decision and + any reasons it gives. Report failed tool calls as failures, and don't + claim to have called a tool you didn't use. If you couldn't find out, say what you tried and what is missing. - id: approve diff --git a/ui/AGENTS.md b/ui/AGENTS.md index d6ca24894..79d7d44bb 100644 --- a/ui/AGENTS.md +++ b/ui/AGENTS.md @@ -79,7 +79,8 @@ state; the UI presents them and submits operator choices. - Name things after what you are working with. The page sets context, so use "Restore default" not "Restore shipped workflows". - No filler or guidance text. Delete empty-state essays, hints that repeat the - label, and sentences explaining how the UI works. + label, and sentences explaining how the UI works. Don't say that secret + values are hidden when the control only displays credential names. - Product names are our own. The editor helper is "Build with Archie" or "Iterate with Archie", never generic terms like "Ask bar". - Actions live where their scope lives. Bulk or instance-wide actions (e.g., diff --git a/ui/src/archie/ActionTrailView.vue b/ui/src/archie/ActionTrailView.vue index bb6a87d2a..96998f6f3 100644 --- a/ui/src/archie/ActionTrailView.vue +++ b/ui/src/archie/ActionTrailView.vue @@ -7,7 +7,6 @@ import { MetadataList } from "@/components/ui/metadata-list"; import { Timestamp } from "@/components/ui/timestamp"; import type { ActionTrail, ApprovalRequest, Effect, Verdict } from "./api"; -/** One action from judgement to effect: verdicts, the approval it was put to, and what it did. */ const props = defineProps<{ action: ActionTrail }>(); type Tone = "ok" | "warn" | "danger" | "info" | "secondary"; diff --git a/ui/src/archie/AgentModelEditor.vue b/ui/src/archie/AgentModelEditor.vue index ed846dedd..63cda5a22 100644 --- a/ui/src/archie/AgentModelEditor.vue +++ b/ui/src/archie/AgentModelEditor.vue @@ -1,5 +1,5 @@