dev: replace the Docker Postgres in task dev with e2e/stack.sh's local cluster #1624

Open
opened 2026-10-27 13:41:08 +00:00 by sam · 1 comment
Owner

task dev still starts PostgreSQL with docker compose while e2e/stack.sh runs a local PostgreSQL 18 without Docker. Two ways to get a dev database. Done when task dev uses the local cluster and the compose postgres service is deleted.


Imported from beads archie-core-v8ucq · GitHub #1657 · type task · priority P3 · created 2026-10-11

task dev still starts PostgreSQL with docker compose while e2e/stack.sh runs a local PostgreSQL 18 without Docker. Two ways to get a dev database. Done when task dev uses the local cluster and the compose postgres service is deleted. --- _Imported from beads `archie-core-v8ucq` · GitHub #1657 · type task · priority P3 · created 2026-10-11_
sam added the lowtask labels 2026-10-27 13:41:08 +00:00
sam self-assigned this 2026-10-27 13:41:08 +00:00
Author
Owner

archie's PRD is ready -- awaiting your go/no-go.

PRD — task dev uses the local PostgreSQL cluster; the compose postgres service is deleted

Issue #1624 (archie-core-v8ucq), priority P3/low. Two corrections to the assessment are marked [corrected]; everything else verified against the tree.

Problem

task dev needs a database and gets it from Docker: Taskfile.yml:145 dev:postgres runs docker compose up -d postgres and polls docker inspect ... .State.Health.Status for 60s. e2e/stack.sh gets the same database without Docker — start_pg requires PostgreSQL 18 binaries (pgbin(): /usr/lib/postgresql/18/bin or postgres on PATH, version-checked), runs initdb -U archie --auth=trust, pg_ctl -w (which already waits for readiness), and createdb archie on ${ARCHIE_DEV_PGPORT:-55432}. Two ways to reach one database, one of them slower and dependent on an image pull. Nothing is broken; this is duplication removal, which AGENTS.md asks for directly ("Delete before adding", "Cheapest option first").

The issue's scope is understated in one way and overstated in another.

  • Understated: deleting the service also invalidates four live references, not just the dev target — deployments/standalone.toml:18, single-forge-github.toml:11, docker-nats-stack.toml:19 and deployments/README.md:45 each promise `docker compose up -d postgres` provides this default.
  • [corrected] Overstated: config.example.toml:10 does not reference the compose service — it is only a commented 5432 example URL and needs no edit. Historical mentions in CHANGELOG.md:854 and docs/news/releases.json:205 must stay as written.
  • Also confirmed safe: install.sh:275-310 writes its own ${ARCHIE_DATA_DIR}/postgres.yml and runs docker compose -p "$INSTANCE" -f "$PG_COMPOSE" up -d --wait, and tools/canary-rehearse.sh:84 starts its own postgres:18 container. No .github/workflow touches postgres. Removing the repo service cannot affect install or the canary rehearsal.

Decisions the PRD settles (with recommendations)

D1 — share the cluster, or share only the code? Recommend: share the cluster (~/.local/state/archie-dev/pg, database archie, port 55432 — the one e2e/stack.sh already makes).

  • Why: it is the smallest change that satisfies the issue ("one dev database"), and start_pg is already idempotent (skips initdb when PG_VERSION exists, skips start when pg_ctl status succeeds). The alternative costs a new shared script plus a dev-only port.
  • Accepted costs, to be stated in the CHANGELOG: dev and e2e share data, so task e2e:up seeds the E2E operator (operator@example.test) and five test tasks into the dev database, dev's tasks appear in e2e, and bash e2e/stack.sh reset (rm -rf "$dir" "$pgdata") deletes dev's data. That is acceptable for a disposable dev database under "breaking changes need no compatibility shims", but it is the operator's call, not the implementer's.
  • Concurrency is unchanged: the two stacks already cannot coexist (e2e/stack.sh start refuses occupied 9091/8585/8586/8587/8485/8484/5173, which deployments/dev.toml also binds).
  • If the operator prefers isolation, D1-alt: move pgbin/pgdata/start_pg/stop_pg into a shared script (e.g. tools/dev-pg.sh, sourced by e2e/stack.sh), give dev its own data dir and port (55433) and keep deployments/dev.toml's isolation. Cost: ~40 more lines and a second port. It also keeps the e2e harness ("never part of the gate") out of the dev path, which is its main merit.

D2 — what the deployment templates say instead. Recommend: delete the service and rewrite the comments to "install PostgreSQL 18 and point database_url at it (task dev starts a local cluster; archied setup offers a bundled one)". Keeping a Docker database in docker-compose.yml only for the template path would turn the issue's "compose postgres service is deleted" into "removed from task dev only" and leave the duplication in place.

Proposed solution

  1. e2e/stack.sh: extract the pg stop block from stop() into stop_pg(), then add start-pg and stop-pg subcommands (plus the usage string). start/stop call the same functions, so behaviour is unchanged.
    • Trap to respect: the script holds flock -n 9 on $dir/stack.lock for its whole run, and start() passes 9>&- to start_pg because the daemonised postgres would otherwise inherit fd 9 and hold the lock forever. The new cases must close fd 9 too (start-pg) start_pg 9>&- ;;), or every later e2e/stack.sh invocation dies with "Another e2e stack command is running".
    • The start-pg path must not run prepare(): that copies config, rewrites the overlay and calls task dev:tls / task build, which would make task dev → stack.sh → task a cycle.
  2. Taskfile.yml:145-158: replace the docker command and health loop with one call to the new subcommand. Keep the target name dev:postgres (dev:state-store at :185 depends on it and nothing else references it) and keep the desc.
  3. deployments/dev.toml:12-17: point database_url at postgres://archie@127.0.0.1:55432/archie?sslmode=disable (matching stack.sh's generated overlay; the cluster is --auth=trust, so the :archie password is misleading) and rewrite the comment that says it must reach the compose service.
  4. docker-compose.yml: delete the postgres service (:18-46), the pg_data volume (:114), the header line # State Store DB: docker compose up -d postgres (:9) and the PostgreSQL for the State Store clause at :1. Keep nats, agent, agent-net, nats_data and the pinned subnet untouched.
  5. Docs/templates: the four live docker compose up -d postgres claims; AGENTS.md:165 prerequisites (add a PostgreSQL 18 server; Docker stays required — agent containers and dev.toml's 172.17.0.1:9091 bridge target); README.md:147-151 Develop section; e2e/README.md (the reset row must now say it deletes the dev database too, and note start-pg/stop-pg).
  6. CHANGELOG.md under [Unreleased]: one bullet naming the host prerequisite, the port change from 5432 to 55432, and that the old pg_data volume's contents are abandoned (the new cluster starts empty and archie-state-store migrates it on first boot). No docs/news regeneration needed — newsgen skips Unreleased (tools/newsgen/changelog.go:50).

Known residual gap to state, not fix: ARCHIE_DEV_PGPORT still overrides the port for stack.sh only, so a non-default port leaves dev.toml pointing at 55432. Document it in e2e/README.md rather than adding a mechanism.

Files and components affected

Taskfile.yml; docker-compose.yml; e2e/stack.sh, e2e/README.md; deployments/dev.toml, deployments/standalone.toml, deployments/single-forge-github.toml, deployments/docker-nats-stack.toml, deployments/README.md; AGENTS.md; README.md; CHANGELOG.md. Unchanged: all Go code, install.sh, tools/canary-rehearse.sh, tools/cloud-setup.sh (already installs postgresql-18 "for e2e/stack.sh", so the cloud dev box is covered), e2e/seed.sh, e2e/run.cjs, CI.

Acceptance criteria

  1. On a host with PostgreSQL 18 server binaries and no compose postgres, task dev reaches a working cluster: psql -h 127.0.0.1 -p 55432 -U archie archie -c '\dt' lists the migrated tables.
  2. task dev:postgres twice in a row succeeds; the second is a no-op (pg_ctl status already running).
  3. Immediately after task dev:postgres, bash e2e/stack.sh status succeeds — proves fd 9 is not held by postgres.
  4. bash e2e/stack.sh stop-pg stops the cluster and start-pg restarts it; task e2e:up / task e2e:down still complete a round trip on the same cluster.
  5. docker compose config -q succeeds, docker compose up -d nats still works, and docker compose up -d postgres now fails with "no such service" (expected).
  6. rg 'docker compose up -d postgres' matches only CHANGELOG.md and docs/news/releases.json; rg 'docker (compose|inspect)' Taskfile.yml matches nothing.
  7. task check passes (no Go change; the e2e stack is outside the gate) and task news:check passes.

Verification is manual only — AGENTS.md forbids wiring tests and no Go code changes, so there is nothing for task test to cover.

Explicit non-goals

  • No task dev:down and no change to dev service lifecycle: Ctrl-C still leaves postgres running, as the compose container did.
  • Do not touch install.sh's bundled PostgreSQL, the agent/agent-net images, or the requirement for Docker (agents plus the 172.17.0.1 bridge target).
  • Do not change e2e/seed.sh, the browser checks, e2e/run.cjs, or add e2e to the gate.
  • No support for PostgreSQL < 18 and no remote/CI database path in task dev.
  • No rewriting of release history in CHANGELOG.md or docs/news.
  • No new config field or port scheme.

Estimated diff size

13 files, roughly +35 / −80, net ≈ −45 lines (below AGENTS.md's ~100-line threshold, and it deletes rather than adds):

File Change
docker-compose.yml −32 (service, volume, 2 header lines)
Taskfile.yml −11 (14-line target → 3)
e2e/stack.sh +8 / −3 (stop_pg extraction, 2 subcommands, usage)
deployments/dev.toml ~5
3 deployment templates + deployments/README.md ~9
AGENTS.md, README.md, e2e/README.md, CHANGELOG.md ~11

Risks and open items

  • Unverified here: whether this working environment has a PostgreSQL 18 server. Discovery was confined to the repository, so the implementer must confirm pgbin finds one before claiming criterion 1 — without it task dev cannot run at all, and that is precisely the new prerequisite.
  • Shared-cluster footguns under D1 (data deleted by reset, E2E fixtures visible in dev) are accepted, not mitigated.
  • Abandoned pg_data volume: developers lose its contents. Stated in the CHANGELOG, no migration shim.
  • Trust auth on a shared localhost cluster means any local process can connect as archie. Already true of the e2e stack; loopback-only, dev-only, not a blocker.
  • Priority low is right: both current database paths work, nothing is lost, no silent failure, no security gap, setup paths keep working. Good filler task; not a reason to interrupt work on the production-ready core.

Decide in chat or on the dashboard.

**archie's PRD is ready -- awaiting your go/no-go.** # PRD — `task dev` uses the local PostgreSQL cluster; the compose postgres service is deleted Issue #1624 (`archie-core-v8ucq`), priority P3/low. Two corrections to the assessment are marked **[corrected]**; everything else verified against the tree. ## Problem `task dev` needs a database and gets it from Docker: `Taskfile.yml:145` `dev:postgres` runs `docker compose up -d postgres` and polls `docker inspect ... .State.Health.Status` for 60s. `e2e/stack.sh` gets the same database without Docker — `start_pg` requires PostgreSQL 18 binaries (`pgbin()`: `/usr/lib/postgresql/18/bin` or `postgres` on PATH, version-checked), runs `initdb -U archie --auth=trust`, `pg_ctl -w` (which already waits for readiness), and `createdb archie` on `${ARCHIE_DEV_PGPORT:-55432}`. Two ways to reach one database, one of them slower and dependent on an image pull. Nothing is broken; this is duplication removal, which AGENTS.md asks for directly ("Delete before adding", "Cheapest option first"). **The issue's scope is understated in one way and overstated in another.** - Understated: deleting the service also invalidates four live references, not just the dev target — `deployments/standalone.toml:18`, `single-forge-github.toml:11`, `docker-nats-stack.toml:19` and `deployments/README.md:45` each promise `` `docker compose up -d postgres` provides this default ``. - **[corrected]** Overstated: `config.example.toml:10` does *not* reference the compose service — it is only a commented 5432 example URL and needs no edit. Historical mentions in `CHANGELOG.md:854` and `docs/news/releases.json:205` must stay as written. - Also confirmed safe: `install.sh:275-310` writes its own `${ARCHIE_DATA_DIR}/postgres.yml` and runs `docker compose -p "$INSTANCE" -f "$PG_COMPOSE" up -d --wait`, and `tools/canary-rehearse.sh:84` starts its own `postgres:18` container. No `.github/workflow` touches postgres. Removing the repo service cannot affect install or the canary rehearsal. ## Decisions the PRD settles (with recommendations) **D1 — share the cluster, or share only the code? Recommend: share the cluster** (`~/.local/state/archie-dev/pg`, database `archie`, port 55432 — the one `e2e/stack.sh` already makes). - Why: it is the smallest change that satisfies the issue ("one dev database"), and `start_pg` is already idempotent (skips `initdb` when `PG_VERSION` exists, skips start when `pg_ctl status` succeeds). The alternative costs a new shared script plus a dev-only port. - Accepted costs, to be stated in the CHANGELOG: dev and e2e share data, so `task e2e:up` seeds the E2E operator (`operator@example.test`) and five test tasks into the dev database, dev's tasks appear in e2e, and `bash e2e/stack.sh reset` (`rm -rf "$dir" "$pgdata"`) deletes dev's data. That is acceptable for a disposable dev database under "breaking changes need no compatibility shims", but it is the operator's call, not the implementer's. - Concurrency is unchanged: the two stacks already cannot coexist (`e2e/stack.sh start` refuses occupied 9091/8585/8586/8587/8485/8484/5173, which `deployments/dev.toml` also binds). - If the operator prefers isolation, **D1-alt**: move `pgbin`/`pgdata`/`start_pg`/`stop_pg` into a shared script (e.g. `tools/dev-pg.sh`, sourced by `e2e/stack.sh`), give dev its own data dir and port (55433) and keep `deployments/dev.toml`'s isolation. Cost: ~40 more lines and a second port. It also keeps the e2e harness ("never part of the gate") out of the dev path, which is its main merit. **D2 — what the deployment templates say instead. Recommend: delete the service and rewrite the comments** to "install PostgreSQL 18 and point `database_url` at it (`task dev` starts a local cluster; `archied setup` offers a bundled one)". Keeping a Docker database in `docker-compose.yml` only for the template path would turn the issue's "compose postgres service is deleted" into "removed from `task dev` only" and leave the duplication in place. ## Proposed solution 1. `e2e/stack.sh`: extract the pg stop block from `stop()` into `stop_pg()`, then add `start-pg` and `stop-pg` subcommands (plus the usage string). `start`/`stop` call the same functions, so behaviour is unchanged. - **Trap to respect:** the script holds `flock -n 9` on `$dir/stack.lock` for its whole run, and `start()` passes `9>&-` to `start_pg` because the daemonised postgres would otherwise inherit fd 9 and hold the lock forever. The new cases must close fd 9 too (`start-pg) start_pg 9>&- ;;`), or every later `e2e/stack.sh` invocation dies with "Another e2e stack command is running". - The `start-pg` path must **not** run `prepare()`: that copies config, rewrites the overlay and calls `task dev:tls` / `task build`, which would make `task dev` → `stack.sh` → `task` a cycle. 2. `Taskfile.yml:145-158`: replace the docker command and health loop with one call to the new subcommand. Keep the target name `dev:postgres` (`dev:state-store` at `:185` depends on it and nothing else references it) and keep the `desc`. 3. `deployments/dev.toml:12-17`: point `database_url` at `postgres://archie@127.0.0.1:55432/archie?sslmode=disable` (matching `stack.sh`'s generated overlay; the cluster is `--auth=trust`, so the `:archie` password is misleading) and rewrite the comment that says it must reach the compose service. 4. `docker-compose.yml`: delete the `postgres` service (`:18-46`), the `pg_data` volume (`:114`), the header line `# State Store DB: docker compose up -d postgres` (`:9`) and the `PostgreSQL for the State Store` clause at `:1`. Keep `nats`, `agent`, `agent-net`, `nats_data` and the pinned subnet untouched. 5. Docs/templates: the four live `docker compose up -d postgres` claims; `AGENTS.md:165` prerequisites (add a PostgreSQL 18 server; Docker stays required — agent containers and `dev.toml`'s `172.17.0.1:9091` bridge target); `README.md:147-151` Develop section; `e2e/README.md` (the `reset` row must now say it deletes the dev database too, and note `start-pg`/`stop-pg`). 6. `CHANGELOG.md` under `[Unreleased]`: one bullet naming the host prerequisite, the port change from 5432 to 55432, and that the old `pg_data` volume's contents are abandoned (the new cluster starts empty and `archie-state-store` migrates it on first boot). No `docs/news` regeneration needed — newsgen skips `Unreleased` (`tools/newsgen/changelog.go:50`). Known residual gap to state, not fix: `ARCHIE_DEV_PGPORT` still overrides the port for `stack.sh` only, so a non-default port leaves `dev.toml` pointing at 55432. Document it in `e2e/README.md` rather than adding a mechanism. ## Files and components affected `Taskfile.yml`; `docker-compose.yml`; `e2e/stack.sh`, `e2e/README.md`; `deployments/dev.toml`, `deployments/standalone.toml`, `deployments/single-forge-github.toml`, `deployments/docker-nats-stack.toml`, `deployments/README.md`; `AGENTS.md`; `README.md`; `CHANGELOG.md`. Unchanged: all Go code, `install.sh`, `tools/canary-rehearse.sh`, `tools/cloud-setup.sh` (already installs `postgresql-18` "for e2e/stack.sh", so the cloud dev box is covered), `e2e/seed.sh`, `e2e/run.cjs`, CI. ## Acceptance criteria 1. On a host with PostgreSQL 18 server binaries and no compose postgres, `task dev` reaches a working cluster: `psql -h 127.0.0.1 -p 55432 -U archie archie -c '\dt'` lists the migrated tables. 2. `task dev:postgres` twice in a row succeeds; the second is a no-op (`pg_ctl status` already running). 3. Immediately after `task dev:postgres`, `bash e2e/stack.sh status` succeeds — proves fd 9 is not held by postgres. 4. `bash e2e/stack.sh stop-pg` stops the cluster and `start-pg` restarts it; `task e2e:up` / `task e2e:down` still complete a round trip on the same cluster. 5. `docker compose config -q` succeeds, `docker compose up -d nats` still works, and `docker compose up -d postgres` now fails with "no such service" (expected). 6. `rg 'docker compose up -d postgres'` matches only `CHANGELOG.md` and `docs/news/releases.json`; `rg 'docker (compose|inspect)' Taskfile.yml` matches nothing. 7. `task check` passes (no Go change; the e2e stack is outside the gate) and `task news:check` passes. Verification is manual only — AGENTS.md forbids wiring tests and no Go code changes, so there is nothing for `task test` to cover. ## Explicit non-goals - No `task dev:down` and no change to dev service lifecycle: Ctrl-C still leaves postgres running, as the compose container did. - Do not touch `install.sh`'s bundled PostgreSQL, the `agent`/`agent-net` images, or the requirement for Docker (agents plus the `172.17.0.1` bridge target). - Do not change `e2e/seed.sh`, the browser checks, `e2e/run.cjs`, or add e2e to the gate. - No support for PostgreSQL < 18 and no remote/CI database path in `task dev`. - No rewriting of release history in `CHANGELOG.md` or `docs/news`. - No new config field or port scheme. ## Estimated diff size 13 files, roughly **+35 / −80, net ≈ −45 lines** (below AGENTS.md's ~100-line threshold, and it deletes rather than adds): | File | Change | | --- | --- | | `docker-compose.yml` | −32 (service, volume, 2 header lines) | | `Taskfile.yml` | −11 (14-line target → 3) | | `e2e/stack.sh` | +8 / −3 (`stop_pg` extraction, 2 subcommands, usage) | | `deployments/dev.toml` | ~5 | | 3 deployment templates + `deployments/README.md` | ~9 | | `AGENTS.md`, `README.md`, `e2e/README.md`, `CHANGELOG.md` | ~11 | ## Risks and open items - **Unverified here:** whether this working environment has a PostgreSQL 18 server. Discovery was confined to the repository, so the implementer must confirm `pgbin` finds one before claiming criterion 1 — without it `task dev` cannot run at all, and that is precisely the new prerequisite. - Shared-cluster footguns under D1 (data deleted by `reset`, E2E fixtures visible in dev) are accepted, not mitigated. - Abandoned `pg_data` volume: developers lose its contents. Stated in the CHANGELOG, no migration shim. - Trust auth on a shared localhost cluster means any local process can connect as `archie`. Already true of the e2e stack; loopback-only, dev-only, not a blocker. - Priority `low` is right: both current database paths work, nothing is lost, no silent failure, no security gap, setup paths keep working. Good filler task; not a reason to interrupt work on the production-ready core. _Decide in chat or on the dashboard._
sam added the deferred label 2026-11-15 17:26:15 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1624