Files
archie/.scratch/explore-daemon-task-loop.md
T
sam bb2193e5e4 feat: add optional NATS JetStream task distribution
Adds optional NATS-based task distribution alongside the existing
SQLite flow. When [nats] is absent from config, zero behaviour change.

New internal/nats package:
- Subject constants matching PRD §2 (archie.task.{bug,feature,default,bootstrap})
- Client: Connect (stream + durable pull consumer), PublishTask
  (Nats-Msg-Id dedup), Fetch, DecodeTask
- Tests with embedded NATS server (connect/publish, dedup, subject routing)

Config: NATSConfig{URL, TokenEnv} on Config, commented-out [nats] block
in config.example.toml

Store: ClaimByIssue atomic claim by owner/repo/number for NATS consumer

Daemon: Nats client field, poll() branches to pollNATS/pollSQLite,
Cycle() branches to drainNATS/drainSQLite (NATS fetch then SQLite
ClaimNext fallback for requeued tasks), processNATSTask
(decode → EnqueueIssue → ClaimByIssue → process → Ack)

Scratch: explorer and plan agent outputs from parallel-explore phase
2026-07-22 02:02:09 +10:00

350 lines
19 KiB
Markdown

# daemon-task-loop
Here is the full report.
---
## 1. Full Lifecycle: GitHub Issue to Completed Task
The lifecycle spans four subsystems: **forge** (GitHub polling), **store** (SQLite persistence), **daemon** (orchestration loop), and **workflow** (pipeline engine). Here is the exact call chain.
### Phase 1 -- Discovery (polling)
**Entry:** `Daemon.Run()` at `daemon.go:72`
- Creates a `time.NewTicker` from `d.Cfg.PollInterval` (default 60s, line 259 of config.go).
- Calls `d.Cycle(ctx)` on every tick.
**`Daemon.Cycle()`** at `daemon.go:88` does three discovery activities, then drains the queue:
1. **`d.poll(ctx)`** -- line 89. Iterates over `d.Cfg.Repos`, calling `d.pollIssues(ctx, repo)` (line 140).
2. **`d.reconcilePRs(ctx)`** -- line 90. Finds tasks at `StatusPROpen` and checks GitHub for merged/closed status.
3. **`d.checkWaiting(ctx)`** -- line 91. Finds tasks at `StatusWaitingHuman` and LLM-judges human replies.
**`Daemon.pollIssues()`** at `daemon.go:140` routes by `d.Cfg.Dispatch.Trigger`:
- `"assignee"`: calls `d.Forge.AssignedIssues(ctx, owner, repo, botUser)`.
- `"label"`: calls `d.Forge.IssuesWithLabel(ctx, owner, repo, label)`.
- `"either"`: calls `pollEither()` (line 163), which unions both lists deduped by issue number.
### Phase 2 -- Enqueue
**`Daemon.poll()`** at `daemon.go:111-133`: for each discovered `forge.Issue`, calls:
- `d.Store.EnqueueIssue(ctx, owner, repo, number, title, body, labels)` -- this is an `INSERT ... ON CONFLICT DO NOTHING` at `store.go:115-118`. Returns `true` if a new row was inserted (idempotency via the `UNIQUE(owner, repo, issue_number)` constraint at line 99).
- On insert: posts an emoji reaction via `d.Forge.React()` (default "eyes", config line 319), sets the "queued" state label, and emits a `KindTaskQueued` event.
- On duplicate (inserted==false): calls `d.maybeRetryParked()` (line 134) which checks whether a human removed the `archie:parked` label and if so calls `d.Store.Requeue()`.
### Phase 3 -- Claim (queue drain)
Back in `Daemon.Cycle()` at lines 92-106, an inner `for` loop:
1. Calls `d.Store.ClaimNext(ctx)` (line 96).
2. If nil, returns (queue empty).
3. Otherwise calls `d.process(ctx, task)` (line 104), then loops again.
**`Store.ClaimNext()`** at `store.go:129-141`: one atomic SQL statement:
```sql
UPDATE tasks SET status='running', attempt=attempt+1, updated_at=datetime('now')
WHERE id = (SELECT id FROM tasks WHERE status='queued' ORDER BY id LIMIT 1)
RETURNING id, owner, repo, issue_number, title, body, labels, status,
workflow, stage, branch, plan, notes, pr_number, tokens_used,
iterations, attempt, park_reason
```
Because SQLite locks the database file for the duration of the UPDATE, no two claimed tasks can overlap -- the daemon is effectively single-consumer. The `RETURNING` clause returns the full row.
### Phase 4 -- Process
**`Daemon.process()`** at `daemon.go:356`:
1. Looks up the `config.Repo` for the task via `d.repoFor(task)` (line 358). Fails/tasks that belong to a now-unconfigured repo go to `StatusParked`.
2. Routes to a workflow: `wf := workflow.Route(task, d.Workflows)` (line 362).
3. Sets the "working" state label on the issue.
4. Calls `workflow.Run(ctx, wf, taskContext)` (line 365).
### Phase 5 -- Workflow Execution
**`workflow.Route()`** at `workflow.go:103-141` decides the workflow:
1. If `t.Workflow` is non-empty (set by a prior `Requeue`), use that exact workflow from the registry.
2. Otherwise inspect labels: `"bug"` -> `"tdd"`, `"feature"` -> `"feasibility"`, `"bootstrap"` -> `"bootstrap"`.
3. Falls back to `"implement"`, then `"default"`, then a hardcoded failing error workflow.
**`workflow.Run()`** at `workflow.go:145-185` executes stages sequentially:
1. Sets `t.Workflow = wf.Name`, persists via `tc.Store.Update()`.
2. For each stage: sets `t.Stage`, persists, emits `KindStageStart`, runs `stage.Run(ctx, tc)`, emits `KindStageFinish` with timing data.
3. On stage error: if `ctx.Err()` is set (daemon shutting down), returns cleanly; otherwise calls `park()` at line 216, which transitions to `StatusParked`, posts a comment, and emits `KindParked`.
4. If `tc.Outcome.Status` is non-empty after a stage, calls `finish()` at line 187, which transitions to that terminal status.
### Phase 6 -- Terminal States
A workflow terminates through `finish()` (`workflow.go:187`) or `park()` (`workflow.go:216`):
| Outcome | Status | What happens |
|---|---|---|
| `park()` | `parked` | Worktree kept, park-reason comment posted, label set. Human can remove `archie:parked` to trigger `maybeRetryParked()`. |
| `finish()` with `StatusPROpen` | `pr_open` | PR is open on GitHub. Subsequent `Cycle()` passes call `reconcilePRs()` which checks merged/closed. |
| `finish()` with `StatusWaitingHuman` | `waiting_human` | Daemon watches for human replies via `checkWaiting()`, LLM-judges approve/reject/unclear. |
| `finish()` with `StatusMerged` | `merged` | Final -- PR merged, labels cleared. |
| `finish()` with `StatusClosedWontDo` | `closed_wont_do` | Issue closed on GitHub. |
| `finish()` with `StatusRejected` | `rejected` | PR closed without merge. |
### Crash Recovery
`Daemon.Startup()` at line 52 calls `d.Store.RecoverStale(ctx)` (`store.go:234`) which runs:
```sql
UPDATE tasks SET status='queued' WHERE status='running'
```
This re-queues any task that was `running` when the previous daemon crashed. There is no distributed locking -- the single SQLite file is the authority.
---
## 2. Event Struct and Bus
### `events.Event` at `events.go:31-42`
```go
type Event struct {
ID int64 `json:"id,omitempty"` // set by the store sink
At time.Time `json:"at"`
Kind string `json:"kind"`
TaskID int64 `json:"task_id,omitempty"`
Repo string `json:"repo,omitempty"`
Issue int `json:"issue,omitempty"`
Workflow string `json:"workflow,omitempty"`
Stage string `json:"stage,omitempty"`
Detail string `json:"detail,omitempty"`
Data map[string]any `json:"data,omitempty"`
}
```
**Kind constants** at `events.go:16-26`:
- `KindTaskQueued`, `KindStageStart`, `KindStageFinish` (data: `duration_ms`, `error`), `KindAgentFinish` (data: `status`, `stop_reason`, `tokens`, `iterations`, `model`), `KindParked` (data: `reason`), `KindOutcome` (data: `status`, `detail`), `KindPRMerged`, `KindPRRejected`, `KindLog` (data: `level`, `msg`).
Events are intentionally the single wire type -- every consumer is just another subscriber.
### `events.Bus` at `events.go:59-128`
A simple in-process fan-out bus:
- **`Subscribe(buffer int) *Sub`** (line 68): creates a buffered channel (default 64), appends to `b.subs`. Returns a `*Sub` whose `.C` field is the read-only receive channel.
- **`Publish(e Event)`** (line 86): stamps `At` if zero, then iterates all subscribers and does a non-blocking send (`select` with `default`). If a subscriber's buffer is full, the event is dropped and `s.dropped.Add(1)` is incremented. **This is the critical design property: backpressure from a slow consumer never blocks the publisher.**
- **`unsubscribe(sub *Sub)`** (line 104): removes the sub from the slice, closes its channel.
- **`Close()`** (line 117): sets `closed` flag, closes all subscriber channels, nils the slice.
- **Bounded per-subscriber buffers** with drop-on-overflow, as documented at line 6: "a stalled dashboard connection can never apply backpressure to the task engine."
### Publish Side
The daemon has a convenience method `emit()` at `daemon.go:44-49`:
```go
func (d *Daemon) emit(e events.Event) {
if d.Bus != nil { d.Bus.Publish(e) }
}
```
Called explicitly at every lifecycle transition -- see `daemon.go:128` (queued), `daemon.go:207` (retried), `daemon.go:237` (PR merged), `daemon.go:248` (PR rejected), `daemon.go:294` (human approved), `daemon.go:303` (human rejected).
The workflow engine also publishes via `TaskContext.Emit()` at `workflow.go:61-75`, which stamps the task identity and delegates to `tc.Bus.Publish()`.
### Subscribe Side (Consumers)
The bus is consumed by at least two subscriber patterns visible in the codebase:
1. **SQLite event log** (`internal/store/events.go`): some goroutine must subscribe and call `s.InsertEvent()` to persist each event. (The subscription setup is in the composition root `cmd/archied`. I did not read that file, but `InsertEvent` at `events.go:32` clearly exists for this purpose.)
2. **SSE fan-out** for the dashboard: the bus is designed so each HTTP SSE connection gets its own `Sub` with a small buffer, and the drop-on-overload behavior lets the dashboard fall behind without slowing the daemon.
---
## 3. ClaimNext -- SQL, Locking, Concurrency
### The SQL
At `store.go:129-141`:
```sql
UPDATE tasks SET status='running', attempt=attempt+1, updated_at=datetime('now')
WHERE id = (SELECT id FROM tasks WHERE status='queued' ORDER BY id LIMIT 1)
RETURNING id, owner, repo, issue_number, title, body, labels, status,
workflow, stage, branch, plan, notes, pr_number, tokens_used,
iterations, attempt, park_reason
```
Three operations in one statement:
- **Subquery**: selects the oldest queued task (`ORDER BY id LIMIT 1`).
- **UPDATE**: atomically marks it `running`, increments `attempt`.
- **RETURNING**: returns the full row, scanned by `scanTask()` (line 143).
### Concurrency Model
- **Single-process, single-threaded consumers.** SQLite's locking means only one writer at a time. The daemon has a single `Store` backed by a `*sql.DB` with `_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)` (line 61). In WAL mode, readers do not block writers, but `UPDATE` still acquires a write lock -- so two concurrent `ClaimNext` calls would serialize on the database lock.
- **No distributed claims.** There is no advisory lock, no `SELECT ... FOR UPDATE`, no external coordination. The claim is just the SQL mutation + the WAL busy_timeout (5s). If two daemon processes somehow connected to the same SQLite file, one would get an error or timeout on concurrent UPDATE -- not a correctness issue, but not a supported deployment.
- **Crash recovery, not fencing.** Instead of distributed locking, `Startup()` calls `RecoverStale()` on boot (`store.go:235`): `UPDATE tasks SET status='queued' WHERE status='running'`. This re-queues any task that was in-flight when the previous process died.
- **Single-consumer queue semantics.** `Daemon.Cycle()` at line 92 loops: claim, process, claim, process. There is no parallelism within one daemon instance -- `process()` is synchronous. The `for` returns when the queue is empty (ClaimNext returns nil).
---
## 4. forge.Issue to store.Task Mapping
### `forge.Issue` at `forge.go:10-15`
```go
type Issue struct {
Number int
Title string
Body string
Labels []string
}
```
A pure value type -- no methods, no identity. It represents one GitHub issue at poll time.
### `store.Task` at `store.go:30-52`
```go
type Task struct {
ID int64 // autoincrement PK
Owner string // GitHub owner
Repo string // GitHub repo
IssueNumber int // matches Issue.Number
Title string // matches Issue.Title
Body string // matches Issue.Body
Labels string // comma-separated from Issue.Labels
Status string // lifecycle: queued, running, waiting_human, ...
Workflow string // routed workflow name
Stage string // current stage name
Branch string // worktree branch
Plan string // PRD / implementation plan
Notes string // appended agent notes
PRNumber int // opened PR number
TokensUsed int
Iterations int
Attempt int
ParkReason string
WatchCommentID int64 // human-reply anchor
}
```
### Mapping at Enqueue Time
In `Daemon.poll()` at `daemon.go:112-113`:
```go
inserted, err := d.Store.EnqueueIssue(ctx,
repo.Owner, repo.Name, is.Number, is.Title, is.Body, strings.Join(is.Labels, ","))
```
The forge-side `[]string` labels are joined into a comma-separated string. The FORGE-side structure is flat (Number, Title, Body, Labels) -- it is the STORE that adds status tracking, routing, and operational fields. The `UNIQUE(owner, repo, issue_number)` constraint at `store.go:99` is the idempotency key: re-polling the same issue does not create a duplicate task.
### Mapping at Enqueue SQL
`store.go:115-118`:
```sql
INSERT INTO tasks (owner, repo, issue_number, title, body, labels)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(owner, repo, issue_number) DO NOTHING
```
Only 6 of 18 Task fields are set at creation. The remaining fields (status defaults to `'queued'`, workflow/stage/branch etc. default to `''` or `0`) are filled in during the processing lifecycle.
---
## 5. What Changes If ClaimNext Is Replaced with NATS JetStream?
### Current Architecture
```
[Ticker] -> Cycle() -> poll() -> [SQLite EnqueueIssue]
-> [SQLite ClaimNext] -> process(task) -> [workflow.Run]
```
The daemon is a single in-memory loop pulling from SQLite. It is **push-based** from SQLite's perspective (the daemon polls the DB), but effectively **single-consumer** -- no parallelism.
### Target Architecture with NATS JetStream
```
[GitHub Webhook] -> [NATS JetStream] -> [Daemon consumes from stream]
```
or
```
[Ticker] -> poll() -> [NATS JetStream Publish]
-> [Daemon Subscribe] -> process(task)
```
### What Stays the Same
1. **`foreach`-over-repos polling** (`Daemon.poll()`, `daemon.go:108`). The GitHub API polling logic (`IssuesWithLabel`, `AssignedIssues`) is independent of the task queue -- you still poll GitHub the same way, you just publish the discovered issue to a NATS subject instead of INSERTing into SQLite.
2. **`workflow.Run()` and the entire workflow engine** (`workflow.go:145-185`, `steps.go`, `agent.go`, `feasibility.go`, `implement.go`, `tdd.go`). The workflow pipeline is agnostic to how tasks arrive -- it receives a `*store.Task` and a `TaskContext`. A NATS consumer would still build the same `TaskContext` and call `workflow.Run()`.
3. **`workflow.Route()`** (`workflow.go:103`). Label-to-workflow routing remains identical -- it operates on `t.Labels` and `t.Workflow`.
4. **`forge` interface and `GitHubClient`** (`forge.go`, `github.go`). The forge is only called to interact with GitHub (comment, label, PR, react) -- not as the task queue. No forge change needed.
5. **`events.Bus` and all `emit()` / `Emit()` calls.** The in-process observability bus is orthogonal to task distribution. NATS JetStream would be the task queue, not the event bus; the two are conceptually separate.
6. **Crash recovery via `RecoverStale()`** would be replaced by NATS's exactly-once delivery (with consumer sequence tracking) or at-least-once with deduplication. But the principle is the same: the daemon should be restartable without losing work.
7. **`store.Update()` and `store.Transition()`** for persisting task progress. Even with NATS as the queue, you still want an audit trail of task state. SQLite is fine for that -- it just becomes the state store rather than the queue.
### What Changes
**A. `Daemon.Run()` and `Cyc1e()` (`daemon.go:72-106`)**
Currently `Run()` has a ticker that calls `Cycle()` which does both poll-and-drain. With NATS:
- The poll ticker would still exist but would `Publish()` discovered issues to a NATS subject rather than `EnqueueIssue`.
- The drain loop would be replaced by a NATS subscription callback. The daemon would register a Pull Consumer or a Push Consumer on the stream, and `process()` would be called from the message handler, not from a `ClaimNext()` call.
Something like:
```go
func (d *Daemon) Run(ctx context.Context) error {
sub, _ := d.JS.PullSubscribe("archie.tasks", ...)
for {
msgs, _ := sub.Fetch(1, ...)
for _, msg := range msgs {
task := deserialize(msg.Data)
d.process(ctx, task)
msg.Ack()
}
}
}
```
**B. `Store.ClaimNext()` is removed (`store.go:129`).**
The atomic SQL `UPDATE ... WHERE id = (SELECT ...)` is the heart of the single-consumer model. With NATS, the message broker provides the claim -- by delivering the message to exactly one consumer in the consumer group (JetStream's built-in `MaxDeliver` / `Ack` / `DeliveryGroup`). The entire function is replaced by `sub.Fetch()` or `sub.Consume()`.
**C. `Store.EnqueueIssue()` (`store.go:114`) becomes optional.**
You may or may not persist tasks in SQLite at enqueue time. Options:
- Keep SQLite as the system of record: the NATS subscriber writes a row on receive, then passes `*store.Task` to `process()`. This preserves the dashboard's ability to query task history without replaying the stream.
- Drop SQLite entirely at the queue level: the NATS message body carries the equivalent of `forge.Issue` + routing metadata, and the first workflow stage does the persistence.
**D. `Daemon.Startup()` crash recovery changes (`daemon.go:52`).**
`RecoverStale()` (`store.go:234`) re-queues tasks left `running`. With NATS:
- If a consumer crashes mid-process, the message is either `NACK`ed or redelivered after the `AckWait` timeout. No explicit recovery step needed.
- But: if NATS delivers a message and the daemon starts processing (updates task to `running` in SQLite), then crashes, the NATS message might already be `Ack`ed. You still need the crash-recovery query as a safety net, or you redesign to only `Ack` after the workflow completes.
- Best practice: `Ack` *after* `workflow.Run()` returns, and keep `RecoverStale` for the SQLite side.
**E. Concurrency model changes fundamentally.**
Currently one daemon processes tasks sequentially (one `ClaimNext` -> `process()` -> next `ClaimNext`). With NATS:
- You can horizontally scale: multiple daemon instances subscribe to the same JetStream consumer group, and NATS round-robins delivery.
- Within one daemon, you can parallelize: call `process()` in a goroutine per message, bounded by a semaphore. Each message has an independent worktree, so parallel workflows on different repos are safe. (Current SQLite model could also parallelize, but the synchronous `Cycle()` design prevents it.)
**F. Idempotency moves from SQL UNIQUE constraint to JetStream dedup.**
Currently `EnqueueIssue` uses `ON CONFLICT DO NOTHING` at `store.go:118`. With NATS, you would use JetStream message deduplication (`Nats-Msg-Id` header set to `owner/repo/issue_number`) to prevent re-processing the same issue enqueue.
**G. Task ordering.**
Currently FIFO via `ORDER BY id LIMIT 1`. NATS JetStream supports ordered delivery within a stream, but with multiple repos and priorities this may differ. You would need a stream per repo or use a subject hierarchy like `archie.tasks.<owner>.<repo>` to preserve ordering semantics.
### Summary Table
| Component | Current (SQLite) | NATS JetStream |
|---|---|---|
| Task queue | `tasks` table, `status='queued'` | JetStream stream |
| Claim | `UPDATE ... WHERE id = (SELECT ...) RETURNING` | `sub.Fetch()` / consumer group |
| Concurrency | Single-process, sequential | Multi-process, parallel via consumer groups |
| Idempotency | `ON CONFLICT DO NOTHING` + `UNIQUE` constraint | `Nats-Msg-Id` dedup |
| Crash recovery | `RecoverStale()` at startup | NAck/redelivery + eventual `RecoverStale` guard |
| Ordering | `ORDER BY id` (FIFO) | Stream ordering per subject |
| Daemon.Run() | `time.Ticker` -> `Cycle()` -> {poll, claim-loop} | `time.Ticker` -> poll() -> publish; sub callback -> process() |
| `Store.ClaimNext()` | **Must remove** | No equivalent needed |
| `Store.EnqueueIssue()` | **Optional** | Still useful as system-of-record |
| Workflow engine | Unchanged | Unchanged |
| Forge interface | Unchanged | Unchanged |
| events.Bus | Unchanged | Unchanged |