Adds optional NATS-based task distribution alongside the existing
SQLite flow. When [nats] is absent from config, zero behaviour change.
New internal/nats package:
- Subject constants matching PRD §2 (archie.task.{bug,feature,default,bootstrap})
- Client: Connect (stream + durable pull consumer), PublishTask
(Nats-Msg-Id dedup), Fetch, DecodeTask
- Tests with embedded NATS server (connect/publish, dedup, subject routing)
Config: NATSConfig{URL, TokenEnv} on Config, commented-out [nats] block
in config.example.toml
Store: ClaimByIssue atomic claim by owner/repo/number for NATS consumer
Daemon: Nats client field, poll() branches to pollNATS/pollSQLite,
Cycle() branches to drainNATS/drainSQLite (NATS fetch then SQLite
ClaimNext fallback for requeued tasks), processNATSTask
(decode → EnqueueIssue → ClaimByIssue → process → Ack)
Scratch: explorer and plan agent outputs from parallel-explore phase
350 lines
19 KiB
Markdown
350 lines
19 KiB
Markdown
# daemon-task-loop
|
|
|
|
Here is the full report.
|
|
|
|
---
|
|
|
|
## 1. Full Lifecycle: GitHub Issue to Completed Task
|
|
|
|
The lifecycle spans four subsystems: **forge** (GitHub polling), **store** (SQLite persistence), **daemon** (orchestration loop), and **workflow** (pipeline engine). Here is the exact call chain.
|
|
|
|
### Phase 1 -- Discovery (polling)
|
|
|
|
**Entry:** `Daemon.Run()` at `daemon.go:72`
|
|
- Creates a `time.NewTicker` from `d.Cfg.PollInterval` (default 60s, line 259 of config.go).
|
|
- Calls `d.Cycle(ctx)` on every tick.
|
|
|
|
**`Daemon.Cycle()`** at `daemon.go:88` does three discovery activities, then drains the queue:
|
|
|
|
1. **`d.poll(ctx)`** -- line 89. Iterates over `d.Cfg.Repos`, calling `d.pollIssues(ctx, repo)` (line 140).
|
|
2. **`d.reconcilePRs(ctx)`** -- line 90. Finds tasks at `StatusPROpen` and checks GitHub for merged/closed status.
|
|
3. **`d.checkWaiting(ctx)`** -- line 91. Finds tasks at `StatusWaitingHuman` and LLM-judges human replies.
|
|
|
|
**`Daemon.pollIssues()`** at `daemon.go:140` routes by `d.Cfg.Dispatch.Trigger`:
|
|
- `"assignee"`: calls `d.Forge.AssignedIssues(ctx, owner, repo, botUser)`.
|
|
- `"label"`: calls `d.Forge.IssuesWithLabel(ctx, owner, repo, label)`.
|
|
- `"either"`: calls `pollEither()` (line 163), which unions both lists deduped by issue number.
|
|
|
|
### Phase 2 -- Enqueue
|
|
|
|
**`Daemon.poll()`** at `daemon.go:111-133`: for each discovered `forge.Issue`, calls:
|
|
- `d.Store.EnqueueIssue(ctx, owner, repo, number, title, body, labels)` -- this is an `INSERT ... ON CONFLICT DO NOTHING` at `store.go:115-118`. Returns `true` if a new row was inserted (idempotency via the `UNIQUE(owner, repo, issue_number)` constraint at line 99).
|
|
- On insert: posts an emoji reaction via `d.Forge.React()` (default "eyes", config line 319), sets the "queued" state label, and emits a `KindTaskQueued` event.
|
|
- On duplicate (inserted==false): calls `d.maybeRetryParked()` (line 134) which checks whether a human removed the `archie:parked` label and if so calls `d.Store.Requeue()`.
|
|
|
|
### Phase 3 -- Claim (queue drain)
|
|
|
|
Back in `Daemon.Cycle()` at lines 92-106, an inner `for` loop:
|
|
1. Calls `d.Store.ClaimNext(ctx)` (line 96).
|
|
2. If nil, returns (queue empty).
|
|
3. Otherwise calls `d.process(ctx, task)` (line 104), then loops again.
|
|
|
|
**`Store.ClaimNext()`** at `store.go:129-141`: one atomic SQL statement:
|
|
```sql
|
|
UPDATE tasks SET status='running', attempt=attempt+1, updated_at=datetime('now')
|
|
WHERE id = (SELECT id FROM tasks WHERE status='queued' ORDER BY id LIMIT 1)
|
|
RETURNING id, owner, repo, issue_number, title, body, labels, status,
|
|
workflow, stage, branch, plan, notes, pr_number, tokens_used,
|
|
iterations, attempt, park_reason
|
|
```
|
|
Because SQLite locks the database file for the duration of the UPDATE, no two claimed tasks can overlap -- the daemon is effectively single-consumer. The `RETURNING` clause returns the full row.
|
|
|
|
### Phase 4 -- Process
|
|
|
|
**`Daemon.process()`** at `daemon.go:356`:
|
|
1. Looks up the `config.Repo` for the task via `d.repoFor(task)` (line 358). Fails/tasks that belong to a now-unconfigured repo go to `StatusParked`.
|
|
2. Routes to a workflow: `wf := workflow.Route(task, d.Workflows)` (line 362).
|
|
3. Sets the "working" state label on the issue.
|
|
4. Calls `workflow.Run(ctx, wf, taskContext)` (line 365).
|
|
|
|
### Phase 5 -- Workflow Execution
|
|
|
|
**`workflow.Route()`** at `workflow.go:103-141` decides the workflow:
|
|
1. If `t.Workflow` is non-empty (set by a prior `Requeue`), use that exact workflow from the registry.
|
|
2. Otherwise inspect labels: `"bug"` -> `"tdd"`, `"feature"` -> `"feasibility"`, `"bootstrap"` -> `"bootstrap"`.
|
|
3. Falls back to `"implement"`, then `"default"`, then a hardcoded failing error workflow.
|
|
|
|
**`workflow.Run()`** at `workflow.go:145-185` executes stages sequentially:
|
|
1. Sets `t.Workflow = wf.Name`, persists via `tc.Store.Update()`.
|
|
2. For each stage: sets `t.Stage`, persists, emits `KindStageStart`, runs `stage.Run(ctx, tc)`, emits `KindStageFinish` with timing data.
|
|
3. On stage error: if `ctx.Err()` is set (daemon shutting down), returns cleanly; otherwise calls `park()` at line 216, which transitions to `StatusParked`, posts a comment, and emits `KindParked`.
|
|
4. If `tc.Outcome.Status` is non-empty after a stage, calls `finish()` at line 187, which transitions to that terminal status.
|
|
|
|
### Phase 6 -- Terminal States
|
|
|
|
A workflow terminates through `finish()` (`workflow.go:187`) or `park()` (`workflow.go:216`):
|
|
|
|
| Outcome | Status | What happens |
|
|
|---|---|---|
|
|
| `park()` | `parked` | Worktree kept, park-reason comment posted, label set. Human can remove `archie:parked` to trigger `maybeRetryParked()`. |
|
|
| `finish()` with `StatusPROpen` | `pr_open` | PR is open on GitHub. Subsequent `Cycle()` passes call `reconcilePRs()` which checks merged/closed. |
|
|
| `finish()` with `StatusWaitingHuman` | `waiting_human` | Daemon watches for human replies via `checkWaiting()`, LLM-judges approve/reject/unclear. |
|
|
| `finish()` with `StatusMerged` | `merged` | Final -- PR merged, labels cleared. |
|
|
| `finish()` with `StatusClosedWontDo` | `closed_wont_do` | Issue closed on GitHub. |
|
|
| `finish()` with `StatusRejected` | `rejected` | PR closed without merge. |
|
|
|
|
### Crash Recovery
|
|
|
|
`Daemon.Startup()` at line 52 calls `d.Store.RecoverStale(ctx)` (`store.go:234`) which runs:
|
|
```sql
|
|
UPDATE tasks SET status='queued' WHERE status='running'
|
|
```
|
|
This re-queues any task that was `running` when the previous daemon crashed. There is no distributed locking -- the single SQLite file is the authority.
|
|
|
|
---
|
|
|
|
## 2. Event Struct and Bus
|
|
|
|
### `events.Event` at `events.go:31-42`
|
|
|
|
```go
|
|
type Event struct {
|
|
ID int64 `json:"id,omitempty"` // set by the store sink
|
|
At time.Time `json:"at"`
|
|
Kind string `json:"kind"`
|
|
TaskID int64 `json:"task_id,omitempty"`
|
|
Repo string `json:"repo,omitempty"`
|
|
Issue int `json:"issue,omitempty"`
|
|
Workflow string `json:"workflow,omitempty"`
|
|
Stage string `json:"stage,omitempty"`
|
|
Detail string `json:"detail,omitempty"`
|
|
Data map[string]any `json:"data,omitempty"`
|
|
}
|
|
```
|
|
|
|
**Kind constants** at `events.go:16-26`:
|
|
- `KindTaskQueued`, `KindStageStart`, `KindStageFinish` (data: `duration_ms`, `error`), `KindAgentFinish` (data: `status`, `stop_reason`, `tokens`, `iterations`, `model`), `KindParked` (data: `reason`), `KindOutcome` (data: `status`, `detail`), `KindPRMerged`, `KindPRRejected`, `KindLog` (data: `level`, `msg`).
|
|
|
|
Events are intentionally the single wire type -- every consumer is just another subscriber.
|
|
|
|
### `events.Bus` at `events.go:59-128`
|
|
|
|
A simple in-process fan-out bus:
|
|
|
|
- **`Subscribe(buffer int) *Sub`** (line 68): creates a buffered channel (default 64), appends to `b.subs`. Returns a `*Sub` whose `.C` field is the read-only receive channel.
|
|
- **`Publish(e Event)`** (line 86): stamps `At` if zero, then iterates all subscribers and does a non-blocking send (`select` with `default`). If a subscriber's buffer is full, the event is dropped and `s.dropped.Add(1)` is incremented. **This is the critical design property: backpressure from a slow consumer never blocks the publisher.**
|
|
- **`unsubscribe(sub *Sub)`** (line 104): removes the sub from the slice, closes its channel.
|
|
- **`Close()`** (line 117): sets `closed` flag, closes all subscriber channels, nils the slice.
|
|
- **Bounded per-subscriber buffers** with drop-on-overflow, as documented at line 6: "a stalled dashboard connection can never apply backpressure to the task engine."
|
|
|
|
### Publish Side
|
|
|
|
The daemon has a convenience method `emit()` at `daemon.go:44-49`:
|
|
```go
|
|
func (d *Daemon) emit(e events.Event) {
|
|
if d.Bus != nil { d.Bus.Publish(e) }
|
|
}
|
|
```
|
|
Called explicitly at every lifecycle transition -- see `daemon.go:128` (queued), `daemon.go:207` (retried), `daemon.go:237` (PR merged), `daemon.go:248` (PR rejected), `daemon.go:294` (human approved), `daemon.go:303` (human rejected).
|
|
|
|
The workflow engine also publishes via `TaskContext.Emit()` at `workflow.go:61-75`, which stamps the task identity and delegates to `tc.Bus.Publish()`.
|
|
|
|
### Subscribe Side (Consumers)
|
|
|
|
The bus is consumed by at least two subscriber patterns visible in the codebase:
|
|
|
|
1. **SQLite event log** (`internal/store/events.go`): some goroutine must subscribe and call `s.InsertEvent()` to persist each event. (The subscription setup is in the composition root `cmd/archied`. I did not read that file, but `InsertEvent` at `events.go:32` clearly exists for this purpose.)
|
|
|
|
2. **SSE fan-out** for the dashboard: the bus is designed so each HTTP SSE connection gets its own `Sub` with a small buffer, and the drop-on-overload behavior lets the dashboard fall behind without slowing the daemon.
|
|
|
|
---
|
|
|
|
## 3. ClaimNext -- SQL, Locking, Concurrency
|
|
|
|
### The SQL
|
|
|
|
At `store.go:129-141`:
|
|
```sql
|
|
UPDATE tasks SET status='running', attempt=attempt+1, updated_at=datetime('now')
|
|
WHERE id = (SELECT id FROM tasks WHERE status='queued' ORDER BY id LIMIT 1)
|
|
RETURNING id, owner, repo, issue_number, title, body, labels, status,
|
|
workflow, stage, branch, plan, notes, pr_number, tokens_used,
|
|
iterations, attempt, park_reason
|
|
```
|
|
|
|
Three operations in one statement:
|
|
- **Subquery**: selects the oldest queued task (`ORDER BY id LIMIT 1`).
|
|
- **UPDATE**: atomically marks it `running`, increments `attempt`.
|
|
- **RETURNING**: returns the full row, scanned by `scanTask()` (line 143).
|
|
|
|
### Concurrency Model
|
|
|
|
- **Single-process, single-threaded consumers.** SQLite's locking means only one writer at a time. The daemon has a single `Store` backed by a `*sql.DB` with `_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)` (line 61). In WAL mode, readers do not block writers, but `UPDATE` still acquires a write lock -- so two concurrent `ClaimNext` calls would serialize on the database lock.
|
|
|
|
- **No distributed claims.** There is no advisory lock, no `SELECT ... FOR UPDATE`, no external coordination. The claim is just the SQL mutation + the WAL busy_timeout (5s). If two daemon processes somehow connected to the same SQLite file, one would get an error or timeout on concurrent UPDATE -- not a correctness issue, but not a supported deployment.
|
|
|
|
- **Crash recovery, not fencing.** Instead of distributed locking, `Startup()` calls `RecoverStale()` on boot (`store.go:235`): `UPDATE tasks SET status='queued' WHERE status='running'`. This re-queues any task that was in-flight when the previous process died.
|
|
|
|
- **Single-consumer queue semantics.** `Daemon.Cycle()` at line 92 loops: claim, process, claim, process. There is no parallelism within one daemon instance -- `process()` is synchronous. The `for` returns when the queue is empty (ClaimNext returns nil).
|
|
|
|
---
|
|
|
|
## 4. forge.Issue to store.Task Mapping
|
|
|
|
### `forge.Issue` at `forge.go:10-15`
|
|
|
|
```go
|
|
type Issue struct {
|
|
Number int
|
|
Title string
|
|
Body string
|
|
Labels []string
|
|
}
|
|
```
|
|
|
|
A pure value type -- no methods, no identity. It represents one GitHub issue at poll time.
|
|
|
|
### `store.Task` at `store.go:30-52`
|
|
|
|
```go
|
|
type Task struct {
|
|
ID int64 // autoincrement PK
|
|
Owner string // GitHub owner
|
|
Repo string // GitHub repo
|
|
IssueNumber int // matches Issue.Number
|
|
Title string // matches Issue.Title
|
|
Body string // matches Issue.Body
|
|
Labels string // comma-separated from Issue.Labels
|
|
Status string // lifecycle: queued, running, waiting_human, ...
|
|
Workflow string // routed workflow name
|
|
Stage string // current stage name
|
|
Branch string // worktree branch
|
|
Plan string // PRD / implementation plan
|
|
Notes string // appended agent notes
|
|
PRNumber int // opened PR number
|
|
TokensUsed int
|
|
Iterations int
|
|
Attempt int
|
|
ParkReason string
|
|
WatchCommentID int64 // human-reply anchor
|
|
}
|
|
```
|
|
|
|
### Mapping at Enqueue Time
|
|
|
|
In `Daemon.poll()` at `daemon.go:112-113`:
|
|
```go
|
|
inserted, err := d.Store.EnqueueIssue(ctx,
|
|
repo.Owner, repo.Name, is.Number, is.Title, is.Body, strings.Join(is.Labels, ","))
|
|
```
|
|
|
|
The forge-side `[]string` labels are joined into a comma-separated string. The FORGE-side structure is flat (Number, Title, Body, Labels) -- it is the STORE that adds status tracking, routing, and operational fields. The `UNIQUE(owner, repo, issue_number)` constraint at `store.go:99` is the idempotency key: re-polling the same issue does not create a duplicate task.
|
|
|
|
### Mapping at Enqueue SQL
|
|
|
|
`store.go:115-118`:
|
|
```sql
|
|
INSERT INTO tasks (owner, repo, issue_number, title, body, labels)
|
|
VALUES (?, ?, ?, ?, ?, ?)
|
|
ON CONFLICT(owner, repo, issue_number) DO NOTHING
|
|
```
|
|
|
|
Only 6 of 18 Task fields are set at creation. The remaining fields (status defaults to `'queued'`, workflow/stage/branch etc. default to `''` or `0`) are filled in during the processing lifecycle.
|
|
|
|
---
|
|
|
|
## 5. What Changes If ClaimNext Is Replaced with NATS JetStream?
|
|
|
|
### Current Architecture
|
|
```
|
|
[Ticker] -> Cycle() -> poll() -> [SQLite EnqueueIssue]
|
|
-> [SQLite ClaimNext] -> process(task) -> [workflow.Run]
|
|
```
|
|
The daemon is a single in-memory loop pulling from SQLite. It is **push-based** from SQLite's perspective (the daemon polls the DB), but effectively **single-consumer** -- no parallelism.
|
|
|
|
### Target Architecture with NATS JetStream
|
|
```
|
|
[GitHub Webhook] -> [NATS JetStream] -> [Daemon consumes from stream]
|
|
```
|
|
or
|
|
```
|
|
[Ticker] -> poll() -> [NATS JetStream Publish]
|
|
-> [Daemon Subscribe] -> process(task)
|
|
```
|
|
|
|
### What Stays the Same
|
|
|
|
1. **`foreach`-over-repos polling** (`Daemon.poll()`, `daemon.go:108`). The GitHub API polling logic (`IssuesWithLabel`, `AssignedIssues`) is independent of the task queue -- you still poll GitHub the same way, you just publish the discovered issue to a NATS subject instead of INSERTing into SQLite.
|
|
|
|
2. **`workflow.Run()` and the entire workflow engine** (`workflow.go:145-185`, `steps.go`, `agent.go`, `feasibility.go`, `implement.go`, `tdd.go`). The workflow pipeline is agnostic to how tasks arrive -- it receives a `*store.Task` and a `TaskContext`. A NATS consumer would still build the same `TaskContext` and call `workflow.Run()`.
|
|
|
|
3. **`workflow.Route()`** (`workflow.go:103`). Label-to-workflow routing remains identical -- it operates on `t.Labels` and `t.Workflow`.
|
|
|
|
4. **`forge` interface and `GitHubClient`** (`forge.go`, `github.go`). The forge is only called to interact with GitHub (comment, label, PR, react) -- not as the task queue. No forge change needed.
|
|
|
|
5. **`events.Bus` and all `emit()` / `Emit()` calls.** The in-process observability bus is orthogonal to task distribution. NATS JetStream would be the task queue, not the event bus; the two are conceptually separate.
|
|
|
|
6. **Crash recovery via `RecoverStale()`** would be replaced by NATS's exactly-once delivery (with consumer sequence tracking) or at-least-once with deduplication. But the principle is the same: the daemon should be restartable without losing work.
|
|
|
|
7. **`store.Update()` and `store.Transition()`** for persisting task progress. Even with NATS as the queue, you still want an audit trail of task state. SQLite is fine for that -- it just becomes the state store rather than the queue.
|
|
|
|
### What Changes
|
|
|
|
**A. `Daemon.Run()` and `Cyc1e()` (`daemon.go:72-106`)**
|
|
|
|
Currently `Run()` has a ticker that calls `Cycle()` which does both poll-and-drain. With NATS:
|
|
- The poll ticker would still exist but would `Publish()` discovered issues to a NATS subject rather than `EnqueueIssue`.
|
|
- The drain loop would be replaced by a NATS subscription callback. The daemon would register a Pull Consumer or a Push Consumer on the stream, and `process()` would be called from the message handler, not from a `ClaimNext()` call.
|
|
|
|
Something like:
|
|
```go
|
|
func (d *Daemon) Run(ctx context.Context) error {
|
|
sub, _ := d.JS.PullSubscribe("archie.tasks", ...)
|
|
for {
|
|
msgs, _ := sub.Fetch(1, ...)
|
|
for _, msg := range msgs {
|
|
task := deserialize(msg.Data)
|
|
d.process(ctx, task)
|
|
msg.Ack()
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
**B. `Store.ClaimNext()` is removed (`store.go:129`).**
|
|
|
|
The atomic SQL `UPDATE ... WHERE id = (SELECT ...)` is the heart of the single-consumer model. With NATS, the message broker provides the claim -- by delivering the message to exactly one consumer in the consumer group (JetStream's built-in `MaxDeliver` / `Ack` / `DeliveryGroup`). The entire function is replaced by `sub.Fetch()` or `sub.Consume()`.
|
|
|
|
**C. `Store.EnqueueIssue()` (`store.go:114`) becomes optional.**
|
|
|
|
You may or may not persist tasks in SQLite at enqueue time. Options:
|
|
- Keep SQLite as the system of record: the NATS subscriber writes a row on receive, then passes `*store.Task` to `process()`. This preserves the dashboard's ability to query task history without replaying the stream.
|
|
- Drop SQLite entirely at the queue level: the NATS message body carries the equivalent of `forge.Issue` + routing metadata, and the first workflow stage does the persistence.
|
|
|
|
**D. `Daemon.Startup()` crash recovery changes (`daemon.go:52`).**
|
|
|
|
`RecoverStale()` (`store.go:234`) re-queues tasks left `running`. With NATS:
|
|
- If a consumer crashes mid-process, the message is either `NACK`ed or redelivered after the `AckWait` timeout. No explicit recovery step needed.
|
|
- But: if NATS delivers a message and the daemon starts processing (updates task to `running` in SQLite), then crashes, the NATS message might already be `Ack`ed. You still need the crash-recovery query as a safety net, or you redesign to only `Ack` after the workflow completes.
|
|
- Best practice: `Ack` *after* `workflow.Run()` returns, and keep `RecoverStale` for the SQLite side.
|
|
|
|
**E. Concurrency model changes fundamentally.**
|
|
|
|
Currently one daemon processes tasks sequentially (one `ClaimNext` -> `process()` -> next `ClaimNext`). With NATS:
|
|
- You can horizontally scale: multiple daemon instances subscribe to the same JetStream consumer group, and NATS round-robins delivery.
|
|
- Within one daemon, you can parallelize: call `process()` in a goroutine per message, bounded by a semaphore. Each message has an independent worktree, so parallel workflows on different repos are safe. (Current SQLite model could also parallelize, but the synchronous `Cycle()` design prevents it.)
|
|
|
|
**F. Idempotency moves from SQL UNIQUE constraint to JetStream dedup.**
|
|
|
|
Currently `EnqueueIssue` uses `ON CONFLICT DO NOTHING` at `store.go:118`. With NATS, you would use JetStream message deduplication (`Nats-Msg-Id` header set to `owner/repo/issue_number`) to prevent re-processing the same issue enqueue.
|
|
|
|
**G. Task ordering.**
|
|
|
|
Currently FIFO via `ORDER BY id LIMIT 1`. NATS JetStream supports ordered delivery within a stream, but with multiple repos and priorities this may differ. You would need a stream per repo or use a subject hierarchy like `archie.tasks.<owner>.<repo>` to preserve ordering semantics.
|
|
|
|
### Summary Table
|
|
|
|
| Component | Current (SQLite) | NATS JetStream |
|
|
|---|---|---|
|
|
| Task queue | `tasks` table, `status='queued'` | JetStream stream |
|
|
| Claim | `UPDATE ... WHERE id = (SELECT ...) RETURNING` | `sub.Fetch()` / consumer group |
|
|
| Concurrency | Single-process, sequential | Multi-process, parallel via consumer groups |
|
|
| Idempotency | `ON CONFLICT DO NOTHING` + `UNIQUE` constraint | `Nats-Msg-Id` dedup |
|
|
| Crash recovery | `RecoverStale()` at startup | NAck/redelivery + eventual `RecoverStale` guard |
|
|
| Ordering | `ORDER BY id` (FIFO) | Stream ordering per subject |
|
|
| Daemon.Run() | `time.Ticker` -> `Cycle()` -> {poll, claim-loop} | `time.Ticker` -> poll() -> publish; sub callback -> process() |
|
|
| `Store.ClaimNext()` | **Must remove** | No equivalent needed |
|
|
| `Store.EnqueueIssue()` | **Optional** | Still useful as system-of-record |
|
|
| Workflow engine | Unchanged | Unchanged |
|
|
| Forge interface | Unchanged | Unchanged |
|
|
| events.Bus | Unchanged | Unchanged | |