Test-run an unsaved workflow draft against a sample event, with forge side effects suppressed #430

Closed
opened 2026-10-27 13:36:18 +00:00 by sam · 0 comments
Owner

Purpose

An operator changing a workflow on the canvas cannot find out whether it works without saving it over the live definition and waiting for a real event. They must be able to test a draft: run the unsaved definition once, against a chosen sample (a captured event or hand-written inputs), and watch it on the canvas, without the draft replacing the stored workflow or being picked up by real traffic.

Today

  • Runs always use a stored definition: dispatch pins the stored YAML and its digest onto the task (CompilePinnedWorkflow in internal/app/agentworker/task_execution.go, task.WorkflowDefinitionYAML/Digest). Run on the canvas is disabled while there are unsaved changes for this reason.
  • A work request (POST /api/work-requests, internal/webui/api_workflows.go workRequest) names a workflow id, repository, title, instructions, inputs.
  • The pinning path already carries a full definition YAML per task, so a task can run a definition that is not the stored one, if the request is allowed to supply it.

Build

  1. A test run is a work request that carries the draft YAML. The server validates it with the same vocabulary as a save, pins it onto the task, and marks the task as a test run (so it is listed apart, never triggers forge side effects you did not choose, and never counts in workflow stats).
  2. Decide and enforce forge safety: by default a test run must not push, open PRs, comment or close issues. The cheapest rule: forge and push steps run in a no-op mode that records what they would have done. Record the decision in the bead.
  3. Canvas: "Test" beside Run, enabled when the draft is valid; it asks for inputs (and optionally a captured event), starts the test run and watches it on the canvas with the would-have-done forge actions shown on their steps.

Context every agent needs

  • The Workflows page (ui/src/workflows/WorkflowsPage.vue, WorkflowEditor.vue) is archie-core's main operator surface: a Vue Flow canvas (WorkflowCanvas.vue, nodes in StepNode.vue, layout in workflow-graph.ts) that both edits a workflow definition and watches its runs. Edits are edits to the YAML document (workflow-edit.ts), so the canvas and the YAML tab never disagree. Read WORKFLOWS.md (repo root) for the workflow language.
  • The watched run comes from useWorkflowRuns (workflow-runs.ts): GET /api/tasks filtered by workflow, and GET /api/tasks/{id}/attempts for its recorded steps (stage rows: name, status ok|failed|interrupted|running, duration_ms, error). A step is recorded under its id, else its type. Live refresh is useLiveResource("tasks", ...) over the SSE stream.
  • Slide-in panels over the right edge of the canvas (StepEditor.vue, WorkflowSettings.vue) are the pattern for details and editing. Do not add a permanent side panel.
  • UI bar the maintainer has set (memory: canvas UI taste): direct manipulation (right-click menus, inline + points, panels that open on demand), no instructional filler text, the field that matters first, help text on hover, nothing that makes the page scroll when the canvas could pan. Drive every change in a headless browser (Playwright with the system Chrome works; stub /api/** with page.route) and look at screenshots at 1920x947 before calling it done.
  • AGENTS.md rules apply: cheapest change first, services own contracts (the dashboard reaches other services only through archie-ui's HTTP API and their gRPC contracts), no new binaries, regenerate and commit ui/dist with the change, task check gates the commit, tests only for the seams AGENTS.md lists.

Acceptance criteria

  • An unsaved draft can be test-run; the stored definition is unchanged and real triggers keep using it.
  • A test run performs no push, PR, comment or issue close; the canvas shows what each forge step would have done.
  • Test runs are visibly distinct in run lists and excluded from workflow statistics.
  • The server rejects an invalid draft exactly as a save would. task check passes; the dispatch/pinning change has a test if it touches a listed seam.

Notes

Decision: forge safety is enforced by suppressing forge and push writes in a test run (recorded as forge_suppressed events on the step); forgerpc and worktreerpc also refuse a test run's credential.

Close reason

Closed


Imported from beads archie-core-t19m.8 · GitHub #1324 · type feature · priority P2 · created 2026-10-05 · closed 2026-10-13

## Purpose An operator changing a workflow on the canvas cannot find out whether it works without saving it over the live definition and waiting for a real event. They must be able to test a draft: run the unsaved definition once, against a chosen sample (a captured event or hand-written inputs), and watch it on the canvas, without the draft replacing the stored workflow or being picked up by real traffic. ## Today - Runs always use a stored definition: dispatch pins the stored YAML and its digest onto the task (`CompilePinnedWorkflow` in `internal/app/agentworker/task_execution.go`, `task.WorkflowDefinitionYAML/Digest`). Run on the canvas is disabled while there are unsaved changes for this reason. - A work request (`POST /api/work-requests`, `internal/webui/api_workflows.go` `workRequest`) names a workflow id, repository, title, instructions, inputs. - The pinning path already carries a full definition YAML per task, so a task can run a definition that is not the stored one, if the request is allowed to supply it. ## Build 1. A test run is a work request that carries the draft YAML. The server validates it with the same vocabulary as a save, pins it onto the task, and marks the task as a test run (so it is listed apart, never triggers forge side effects you did not choose, and never counts in workflow stats). 2. Decide and enforce forge safety: by default a test run must not push, open PRs, comment or close issues. The cheapest rule: forge and push steps run in a no-op mode that records what they would have done. Record the decision in the bead. 3. Canvas: "Test" beside Run, enabled when the draft is valid; it asks for inputs (and optionally a captured event), starts the test run and watches it on the canvas with the would-have-done forge actions shown on their steps. ## Context every agent needs - The Workflows page (`ui/src/workflows/WorkflowsPage.vue`, `WorkflowEditor.vue`) is archie-core's main operator surface: a Vue Flow canvas (`WorkflowCanvas.vue`, nodes in `StepNode.vue`, layout in `workflow-graph.ts`) that both edits a workflow definition and watches its runs. Edits are edits to the YAML document (`workflow-edit.ts`), so the canvas and the YAML tab never disagree. Read `WORKFLOWS.md` (repo root) for the workflow language. - The watched run comes from `useWorkflowRuns` (`workflow-runs.ts`): `GET /api/tasks` filtered by workflow, and `GET /api/tasks/{id}/attempts` for its recorded steps (stage rows: name, status ok|failed|interrupted|running, duration_ms, error). A step is recorded under its `id`, else its type. Live refresh is `useLiveResource("tasks", ...)` over the SSE stream. - Slide-in panels over the right edge of the canvas (`StepEditor.vue`, `WorkflowSettings.vue`) are the pattern for details and editing. Do not add a permanent side panel. - UI bar the maintainer has set (memory: canvas UI taste): direct manipulation (right-click menus, inline + points, panels that open on demand), no instructional filler text, the field that matters first, help text on hover, nothing that makes the page scroll when the canvas could pan. Drive every change in a headless browser (Playwright with the system Chrome works; stub `/api/**` with `page.route`) and look at screenshots at 1920x947 before calling it done. - AGENTS.md rules apply: cheapest change first, services own contracts (the dashboard reaches other services only through archie-ui's HTTP API and their gRPC contracts), no new binaries, regenerate and commit `ui/dist` with the change, `task check` gates the commit, tests only for the seams AGENTS.md lists. ### Acceptance criteria - An unsaved draft can be test-run; the stored definition is unchanged and real triggers keep using it. - A test run performs no push, PR, comment or issue close; the canvas shows what each forge step would have done. - Test runs are visibly distinct in run lists and excluded from workflow statistics. - The server rejects an invalid draft exactly as a save would. `task check` passes; the dispatch/pinning change has a test if it touches a listed seam. ### Notes Decision: forge safety is enforced by suppressing forge and push writes in a test run (recorded as forge_suppressed events on the step); forgerpc and worktreerpc also refuse a test run's credential. ### Close reason Closed --- _Imported from beads `archie-core-t19m.8` · GitHub #1324 · type feature · priority P2 · created 2026-10-05 · closed 2026-10-13_
sam added the mediumfeature labels 2026-10-27 13:36:18 +00:00
sam self-assigned this 2026-10-27 13:36:18 +00:00
sam closed this issue 2026-10-27 13:36:18 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#430