Files
samandCopilot 64dfad6d53 feat(runs): add typed results and model aliases (#2043)
Also fix repository-free message.send validation (#2046) and add a single-binary development smoke path (#1977).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-11-18 23:47:35 +11:00

6.7 KiB

e2e

Validation of the whole stack, run by hand or by an agent, never by task check or CI. It needs PostgreSQL 18, five services, Vite and a browser, which the gate must not. Docker is not needed: without it tasks queue and park, which is useful data for the dashboard.

File What it does
stack.sh Starts PostgreSQL 18 (port 55432) and the five services as local processes, then Vite on 5173
seed.sh Seeds the stack through the dashboard API; safe to rerun
run.cjs Runs checks/*.cjs in Chromium; screenshots, manifest and report.html to out/<stamp>/
checks/ One module per behaviour
results/ One committed report per release

Run

task e2e:up        # build, start everything, seed
task e2e:check                      # list available checks (no stack needed)
task e2e:check -- list              # explicitly list checks
task e2e:check -- audit-redirect    # one check
task e2e:check -- all               # all checks
task e2e:check -- --headed accessibility # watch a check in Chromium
task e2e:check -- --headed all      # watch every check
task e2e:down      # stop; e2e/stack.sh reset also deletes the data

Headless is the default. --headed opens visible Chromium and slows browser actions to 150ms so you can follow them; assertions and saved evidence are the same in both modes. The terminal prints each check before it starts. Headed mode requires a graphical desktop.

Nothing reads the installed config: stack.sh points the XDG directories at ${ARCHIE_DEV_DIR:-~/.local/state/archie-dev} and writes its overlay from deployments/dev.toml. Logs are in its logs/. e2e/stack.sh restart archied restarts one service. Run task e2e:down before starting again: startup refuses occupied service ports and checks that each launched process stays alive.

The seed creates a local test administrator: operator@example.test, password archie-local-e2e-password. Use it at http://localhost:5173; the browser checks sign in with the same account. These are test fixtures for this loopback-only stack. To use an existing test account, set ARCHIE_E2E_EMAIL and ARCHIE_E2E_PASSWORD for both seeding and browser checks. Seeding never resets an existing account or deletes test data.

The first run needs pnpm install --dir e2e and pnpm --dir e2e exec playwright install chromium. In the cloud environment, Chromium and Playwright are preinstalled; run with NODE_PATH=$(pnpm root -g).

Single-binary smoke check

Point this check at an already running, disposable archie instance with its own PostgreSQL database and no external delivery connectors. Set ARCHIE_SMOKE_TOKEN to that instance's bootstrap token and ARCHIE_SMOKE_URL to its IPv4 loopback URL (for example http://127.0.0.1:8080), then run:

node e2e/archie-smoke.cjs

The check saves a workflow and a one-minute schedule, then verifies that its run links back to the schedule event and appears in the composer. It disables the schedule after the check but leaves records for inspection. No model or forge is needed. The legacy task e2e:up stack is not the target of this check. If Archie is running in WSL and its loopback URL is not reachable from Windows, run the check with Node in the same Linux environment as Archie. If the schedule starts but the run does not finish, check the events and runs logs; readiness alone does not verify that either worker can claim work.

Release evidence

node e2e/run.cjs --record v1.54.0 all

writes results/v1.54.0.md, the pass, fail or skip of every check for that release. Commit it with the release. Screenshots stay in out/.

Writing a check

A module in checks/ exports async ({ page, go, shot, fail, skip, settled, attach, tolerate }).

  • go(path) loads a dashboard route: waitUntil: 'load' plus 1200ms. Wait for elements, never for networkidle: the event stream never goes idle.
  • shot(label) saves <check>-<kebab>[-<w>x<h>].png after a short settle, and records {file, w, h, check} in the manifest. The label is slugified, so shot('/') becomes routes-home.png; a viewport other than the 1440x900 default adds a suffix (-390x844). Repeated labels get -2, -3, so every capture is kept.
  • fail(msg) records a problem without stopping.
  • skip(reason) stops the check and records a skip: a missing precondition, never a failure.
  • tolerate(re) ignores console errors matching re, for a response the page is designed to survive.
  • attach(name, data) saves a non-image artifact as <check>-<name>.
  • settled(sel?) waits for a selector or URL, then settles briefly. It is the harness-owned wait; checks need no capture wrapper or ad-hoc sleeps around screenshots.

Each check gets a fresh page at 1440x900 with the login storageState; a check that needs a phone viewport resizes it itself to 390x844. Those are the only two viewports in use. A throw, a fail, or any console or page error fails the check. One module covers one behaviour; families share a prefix: audit-* for the audit page (redirect, filters, revert, runtime, views-export, retention-live, phone), workflow-* for the canvas (loading, validation, phone-layout, logic), chat-composer for chat input, chat-dock-* for the chat dock (pin, drag-cancel, layout, resize, drafts, picker, phone).

routes loads every static route and flags text with stray leading, trailing or doubled spaces. seeded checks the seed is visible. /logs is visited but skipped by the whitespace lint: log lines are data, not copy.

Output

Every run writes out/<stamp>/ with the screenshots, results.json (the manifest: {check, ok, ms, problems, artifacts[]}, plus skipped/skipReason for skips) and report.html: a self-contained test record with the commit, totals and one row per check (status, time, message and that check's artifacts, screenshots inline). attach(name, data) adds a non-image artifact to the current check; a failed check also gets a failure screenshot.

Exits: 0 when every check passes or skips, 1 when any check fails, 2 when the harness itself errors. ARCHIE_E2E_URL points the run at a dashboard other than http://localhost:5173.

Pitfalls

Symptom Cause Fix
State Store exits: server is older than PostgreSQL 18 another server answered stack.sh uses port 55432; set ARCHIE_DEV_PGPORT if that is taken
Startup says a port is in use a stack or another process already owns it Run task e2e:down, then task e2e:up; inspect any remaining listener with ss -ltnp
Your own shell is killed (exit 144) pkill -f matched its own command line stack.sh stop or restart, which kill by PID
Postgres aborts in the cloud /tmp/claude-0 is reset to mode 700 stack.sh keeps the root cluster in /var/lib/archie-dev