Adds optional NATS-based task distribution alongside the existing
SQLite flow. When [nats] is absent from config, zero behaviour change.
New internal/nats package:
- Subject constants matching PRD §2 (archie.task.{bug,feature,default,bootstrap})
- Client: Connect (stream + durable pull consumer), PublishTask
(Nats-Msg-Id dedup), Fetch, DecodeTask
- Tests with embedded NATS server (connect/publish, dedup, subject routing)
Config: NATSConfig{URL, TokenEnv} on Config, commented-out [nats] block
in config.example.toml
Store: ClaimByIssue atomic claim by owner/repo/number for NATS consumer
Daemon: Nats client field, poll() branches to pollNATS/pollSQLite,
Cycle() branches to drainNATS/drainSQLite (NATS fetch then SQLite
ClaimNext fallback for requeued tasks), processNATSTask
(decode → EnqueueIssue → ClaimByIssue → process → Ack)
Scratch: explorer and plan agent outputs from parallel-explore phase
19 KiB
daemon-task-loop
Here is the full report.
1. Full Lifecycle: GitHub Issue to Completed Task
The lifecycle spans four subsystems: forge (GitHub polling), store (SQLite persistence), daemon (orchestration loop), and workflow (pipeline engine). Here is the exact call chain.
Phase 1 -- Discovery (polling)
Entry: Daemon.Run() at daemon.go:72
- Creates a
time.NewTickerfromd.Cfg.PollInterval(default 60s, line 259 of config.go). - Calls
d.Cycle(ctx)on every tick.
Daemon.Cycle() at daemon.go:88 does three discovery activities, then drains the queue:
d.poll(ctx)-- line 89. Iterates overd.Cfg.Repos, callingd.pollIssues(ctx, repo)(line 140).d.reconcilePRs(ctx)-- line 90. Finds tasks atStatusPROpenand checks GitHub for merged/closed status.d.checkWaiting(ctx)-- line 91. Finds tasks atStatusWaitingHumanand LLM-judges human replies.
Daemon.pollIssues() at daemon.go:140 routes by d.Cfg.Dispatch.Trigger:
"assignee": callsd.Forge.AssignedIssues(ctx, owner, repo, botUser)."label": callsd.Forge.IssuesWithLabel(ctx, owner, repo, label)."either": callspollEither()(line 163), which unions both lists deduped by issue number.
Phase 2 -- Enqueue
Daemon.poll() at daemon.go:111-133: for each discovered forge.Issue, calls:
d.Store.EnqueueIssue(ctx, owner, repo, number, title, body, labels)-- this is anINSERT ... ON CONFLICT DO NOTHINGatstore.go:115-118. Returnstrueif a new row was inserted (idempotency via theUNIQUE(owner, repo, issue_number)constraint at line 99).- On insert: posts an emoji reaction via
d.Forge.React()(default "eyes", config line 319), sets the "queued" state label, and emits aKindTaskQueuedevent. - On duplicate (inserted==false): calls
d.maybeRetryParked()(line 134) which checks whether a human removed thearchie:parkedlabel and if so callsd.Store.Requeue().
Phase 3 -- Claim (queue drain)
Back in Daemon.Cycle() at lines 92-106, an inner for loop:
- Calls
d.Store.ClaimNext(ctx)(line 96). - If nil, returns (queue empty).
- Otherwise calls
d.process(ctx, task)(line 104), then loops again.
Store.ClaimNext() at store.go:129-141: one atomic SQL statement:
UPDATE tasks SET status='running', attempt=attempt+1, updated_at=datetime('now')
WHERE id = (SELECT id FROM tasks WHERE status='queued' ORDER BY id LIMIT 1)
RETURNING id, owner, repo, issue_number, title, body, labels, status,
workflow, stage, branch, plan, notes, pr_number, tokens_used,
iterations, attempt, park_reason
Because SQLite locks the database file for the duration of the UPDATE, no two claimed tasks can overlap -- the daemon is effectively single-consumer. The RETURNING clause returns the full row.
Phase 4 -- Process
Daemon.process() at daemon.go:356:
- Looks up the
config.Repofor the task viad.repoFor(task)(line 358). Fails/tasks that belong to a now-unconfigured repo go toStatusParked. - Routes to a workflow:
wf := workflow.Route(task, d.Workflows)(line 362). - Sets the "working" state label on the issue.
- Calls
workflow.Run(ctx, wf, taskContext)(line 365).
Phase 5 -- Workflow Execution
workflow.Route() at workflow.go:103-141 decides the workflow:
- If
t.Workflowis non-empty (set by a priorRequeue), use that exact workflow from the registry. - Otherwise inspect labels:
"bug"->"tdd","feature"->"feasibility","bootstrap"->"bootstrap". - Falls back to
"implement", then"default", then a hardcoded failing error workflow.
workflow.Run() at workflow.go:145-185 executes stages sequentially:
- Sets
t.Workflow = wf.Name, persists viatc.Store.Update(). - For each stage: sets
t.Stage, persists, emitsKindStageStart, runsstage.Run(ctx, tc), emitsKindStageFinishwith timing data. - On stage error: if
ctx.Err()is set (daemon shutting down), returns cleanly; otherwise callspark()at line 216, which transitions toStatusParked, posts a comment, and emitsKindParked. - If
tc.Outcome.Statusis non-empty after a stage, callsfinish()at line 187, which transitions to that terminal status.
Phase 6 -- Terminal States
A workflow terminates through finish() (workflow.go:187) or park() (workflow.go:216):
| Outcome | Status | What happens |
|---|---|---|
park() |
parked |
Worktree kept, park-reason comment posted, label set. Human can remove archie:parked to trigger maybeRetryParked(). |
finish() with StatusPROpen |
pr_open |
PR is open on GitHub. Subsequent Cycle() passes call reconcilePRs() which checks merged/closed. |
finish() with StatusWaitingHuman |
waiting_human |
Daemon watches for human replies via checkWaiting(), LLM-judges approve/reject/unclear. |
finish() with StatusMerged |
merged |
Final -- PR merged, labels cleared. |
finish() with StatusClosedWontDo |
closed_wont_do |
Issue closed on GitHub. |
finish() with StatusRejected |
rejected |
PR closed without merge. |
Crash Recovery
Daemon.Startup() at line 52 calls d.Store.RecoverStale(ctx) (store.go:234) which runs:
UPDATE tasks SET status='queued' WHERE status='running'
This re-queues any task that was running when the previous daemon crashed. There is no distributed locking -- the single SQLite file is the authority.
2. Event Struct and Bus
events.Event at events.go:31-42
type Event struct {
ID int64 `json:"id,omitempty"` // set by the store sink
At time.Time `json:"at"`
Kind string `json:"kind"`
TaskID int64 `json:"task_id,omitempty"`
Repo string `json:"repo,omitempty"`
Issue int `json:"issue,omitempty"`
Workflow string `json:"workflow,omitempty"`
Stage string `json:"stage,omitempty"`
Detail string `json:"detail,omitempty"`
Data map[string]any `json:"data,omitempty"`
}
Kind constants at events.go:16-26:
KindTaskQueued,KindStageStart,KindStageFinish(data:duration_ms,error),KindAgentFinish(data:status,stop_reason,tokens,iterations,model),KindParked(data:reason),KindOutcome(data:status,detail),KindPRMerged,KindPRRejected,KindLog(data:level,msg).
Events are intentionally the single wire type -- every consumer is just another subscriber.
events.Bus at events.go:59-128
A simple in-process fan-out bus:
Subscribe(buffer int) *Sub(line 68): creates a buffered channel (default 64), appends tob.subs. Returns a*Subwhose.Cfield is the read-only receive channel.Publish(e Event)(line 86): stampsAtif zero, then iterates all subscribers and does a non-blocking send (selectwithdefault). If a subscriber's buffer is full, the event is dropped ands.dropped.Add(1)is incremented. This is the critical design property: backpressure from a slow consumer never blocks the publisher.unsubscribe(sub *Sub)(line 104): removes the sub from the slice, closes its channel.Close()(line 117): setsclosedflag, closes all subscriber channels, nils the slice.- Bounded per-subscriber buffers with drop-on-overflow, as documented at line 6: "a stalled dashboard connection can never apply backpressure to the task engine."
Publish Side
The daemon has a convenience method emit() at daemon.go:44-49:
func (d *Daemon) emit(e events.Event) {
if d.Bus != nil { d.Bus.Publish(e) }
}
Called explicitly at every lifecycle transition -- see daemon.go:128 (queued), daemon.go:207 (retried), daemon.go:237 (PR merged), daemon.go:248 (PR rejected), daemon.go:294 (human approved), daemon.go:303 (human rejected).
The workflow engine also publishes via TaskContext.Emit() at workflow.go:61-75, which stamps the task identity and delegates to tc.Bus.Publish().
Subscribe Side (Consumers)
The bus is consumed by at least two subscriber patterns visible in the codebase:
-
SQLite event log (
internal/store/events.go): some goroutine must subscribe and calls.InsertEvent()to persist each event. (The subscription setup is in the composition rootcmd/archied. I did not read that file, butInsertEventatevents.go:32clearly exists for this purpose.) -
SSE fan-out for the dashboard: the bus is designed so each HTTP SSE connection gets its own
Subwith a small buffer, and the drop-on-overload behavior lets the dashboard fall behind without slowing the daemon.
3. ClaimNext -- SQL, Locking, Concurrency
The SQL
At store.go:129-141:
UPDATE tasks SET status='running', attempt=attempt+1, updated_at=datetime('now')
WHERE id = (SELECT id FROM tasks WHERE status='queued' ORDER BY id LIMIT 1)
RETURNING id, owner, repo, issue_number, title, body, labels, status,
workflow, stage, branch, plan, notes, pr_number, tokens_used,
iterations, attempt, park_reason
Three operations in one statement:
- Subquery: selects the oldest queued task (
ORDER BY id LIMIT 1). - UPDATE: atomically marks it
running, incrementsattempt. - RETURNING: returns the full row, scanned by
scanTask()(line 143).
Concurrency Model
-
Single-process, single-threaded consumers. SQLite's locking means only one writer at a time. The daemon has a single
Storebacked by a*sql.DBwith_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)(line 61). In WAL mode, readers do not block writers, butUPDATEstill acquires a write lock -- so two concurrentClaimNextcalls would serialize on the database lock. -
No distributed claims. There is no advisory lock, no
SELECT ... FOR UPDATE, no external coordination. The claim is just the SQL mutation + the WAL busy_timeout (5s). If two daemon processes somehow connected to the same SQLite file, one would get an error or timeout on concurrent UPDATE -- not a correctness issue, but not a supported deployment. -
Crash recovery, not fencing. Instead of distributed locking,
Startup()callsRecoverStale()on boot (store.go:235):UPDATE tasks SET status='queued' WHERE status='running'. This re-queues any task that was in-flight when the previous process died. -
Single-consumer queue semantics.
Daemon.Cycle()at line 92 loops: claim, process, claim, process. There is no parallelism within one daemon instance --process()is synchronous. Theforreturns when the queue is empty (ClaimNext returns nil).
4. forge.Issue to store.Task Mapping
forge.Issue at forge.go:10-15
type Issue struct {
Number int
Title string
Body string
Labels []string
}
A pure value type -- no methods, no identity. It represents one GitHub issue at poll time.
store.Task at store.go:30-52
type Task struct {
ID int64 // autoincrement PK
Owner string // GitHub owner
Repo string // GitHub repo
IssueNumber int // matches Issue.Number
Title string // matches Issue.Title
Body string // matches Issue.Body
Labels string // comma-separated from Issue.Labels
Status string // lifecycle: queued, running, waiting_human, ...
Workflow string // routed workflow name
Stage string // current stage name
Branch string // worktree branch
Plan string // PRD / implementation plan
Notes string // appended agent notes
PRNumber int // opened PR number
TokensUsed int
Iterations int
Attempt int
ParkReason string
WatchCommentID int64 // human-reply anchor
}
Mapping at Enqueue Time
In Daemon.poll() at daemon.go:112-113:
inserted, err := d.Store.EnqueueIssue(ctx,
repo.Owner, repo.Name, is.Number, is.Title, is.Body, strings.Join(is.Labels, ","))
The forge-side []string labels are joined into a comma-separated string. The FORGE-side structure is flat (Number, Title, Body, Labels) -- it is the STORE that adds status tracking, routing, and operational fields. The UNIQUE(owner, repo, issue_number) constraint at store.go:99 is the idempotency key: re-polling the same issue does not create a duplicate task.
Mapping at Enqueue SQL
store.go:115-118:
INSERT INTO tasks (owner, repo, issue_number, title, body, labels)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(owner, repo, issue_number) DO NOTHING
Only 6 of 18 Task fields are set at creation. The remaining fields (status defaults to 'queued', workflow/stage/branch etc. default to '' or 0) are filled in during the processing lifecycle.
5. What Changes If ClaimNext Is Replaced with NATS JetStream?
Current Architecture
[Ticker] -> Cycle() -> poll() -> [SQLite EnqueueIssue]
-> [SQLite ClaimNext] -> process(task) -> [workflow.Run]
The daemon is a single in-memory loop pulling from SQLite. It is push-based from SQLite's perspective (the daemon polls the DB), but effectively single-consumer -- no parallelism.
Target Architecture with NATS JetStream
[GitHub Webhook] -> [NATS JetStream] -> [Daemon consumes from stream]
or
[Ticker] -> poll() -> [NATS JetStream Publish]
-> [Daemon Subscribe] -> process(task)
What Stays the Same
-
foreach-over-repos polling (Daemon.poll(),daemon.go:108). The GitHub API polling logic (IssuesWithLabel,AssignedIssues) is independent of the task queue -- you still poll GitHub the same way, you just publish the discovered issue to a NATS subject instead of INSERTing into SQLite. -
workflow.Run()and the entire workflow engine (workflow.go:145-185,steps.go,agent.go,feasibility.go,implement.go,tdd.go). The workflow pipeline is agnostic to how tasks arrive -- it receives a*store.Taskand aTaskContext. A NATS consumer would still build the sameTaskContextand callworkflow.Run(). -
workflow.Route()(workflow.go:103). Label-to-workflow routing remains identical -- it operates ont.Labelsandt.Workflow. -
forgeinterface andGitHubClient(forge.go,github.go). The forge is only called to interact with GitHub (comment, label, PR, react) -- not as the task queue. No forge change needed. -
events.Busand allemit()/Emit()calls. The in-process observability bus is orthogonal to task distribution. NATS JetStream would be the task queue, not the event bus; the two are conceptually separate. -
Crash recovery via
RecoverStale()would be replaced by NATS's exactly-once delivery (with consumer sequence tracking) or at-least-once with deduplication. But the principle is the same: the daemon should be restartable without losing work. -
store.Update()andstore.Transition()for persisting task progress. Even with NATS as the queue, you still want an audit trail of task state. SQLite is fine for that -- it just becomes the state store rather than the queue.
What Changes
A. Daemon.Run() and Cyc1e() (daemon.go:72-106)
Currently Run() has a ticker that calls Cycle() which does both poll-and-drain. With NATS:
- The poll ticker would still exist but would
Publish()discovered issues to a NATS subject rather thanEnqueueIssue. - The drain loop would be replaced by a NATS subscription callback. The daemon would register a Pull Consumer or a Push Consumer on the stream, and
process()would be called from the message handler, not from aClaimNext()call.
Something like:
func (d *Daemon) Run(ctx context.Context) error {
sub, _ := d.JS.PullSubscribe("archie.tasks", ...)
for {
msgs, _ := sub.Fetch(1, ...)
for _, msg := range msgs {
task := deserialize(msg.Data)
d.process(ctx, task)
msg.Ack()
}
}
}
B. Store.ClaimNext() is removed (store.go:129).
The atomic SQL UPDATE ... WHERE id = (SELECT ...) is the heart of the single-consumer model. With NATS, the message broker provides the claim -- by delivering the message to exactly one consumer in the consumer group (JetStream's built-in MaxDeliver / Ack / DeliveryGroup). The entire function is replaced by sub.Fetch() or sub.Consume().
C. Store.EnqueueIssue() (store.go:114) becomes optional.
You may or may not persist tasks in SQLite at enqueue time. Options:
- Keep SQLite as the system of record: the NATS subscriber writes a row on receive, then passes
*store.Tasktoprocess(). This preserves the dashboard's ability to query task history without replaying the stream. - Drop SQLite entirely at the queue level: the NATS message body carries the equivalent of
forge.Issue+ routing metadata, and the first workflow stage does the persistence.
D. Daemon.Startup() crash recovery changes (daemon.go:52).
RecoverStale() (store.go:234) re-queues tasks left running. With NATS:
- If a consumer crashes mid-process, the message is either
NACKed or redelivered after theAckWaittimeout. No explicit recovery step needed. - But: if NATS delivers a message and the daemon starts processing (updates task to
runningin SQLite), then crashes, the NATS message might already beAcked. You still need the crash-recovery query as a safety net, or you redesign to onlyAckafter the workflow completes. - Best practice:
Ackafterworkflow.Run()returns, and keepRecoverStalefor the SQLite side.
E. Concurrency model changes fundamentally.
Currently one daemon processes tasks sequentially (one ClaimNext -> process() -> next ClaimNext). With NATS:
- You can horizontally scale: multiple daemon instances subscribe to the same JetStream consumer group, and NATS round-robins delivery.
- Within one daemon, you can parallelize: call
process()in a goroutine per message, bounded by a semaphore. Each message has an independent worktree, so parallel workflows on different repos are safe. (Current SQLite model could also parallelize, but the synchronousCycle()design prevents it.)
F. Idempotency moves from SQL UNIQUE constraint to JetStream dedup.
Currently EnqueueIssue uses ON CONFLICT DO NOTHING at store.go:118. With NATS, you would use JetStream message deduplication (Nats-Msg-Id header set to owner/repo/issue_number) to prevent re-processing the same issue enqueue.
G. Task ordering.
Currently FIFO via ORDER BY id LIMIT 1. NATS JetStream supports ordered delivery within a stream, but with multiple repos and priorities this may differ. You would need a stream per repo or use a subject hierarchy like archie.tasks.<owner>.<repo> to preserve ordering semantics.
Summary Table
| Component | Current (SQLite) | NATS JetStream |
|---|---|---|
| Task queue | tasks table, status='queued' |
JetStream stream |
| Claim | UPDATE ... WHERE id = (SELECT ...) RETURNING |
sub.Fetch() / consumer group |
| Concurrency | Single-process, sequential | Multi-process, parallel via consumer groups |
| Idempotency | ON CONFLICT DO NOTHING + UNIQUE constraint |
Nats-Msg-Id dedup |
| Crash recovery | RecoverStale() at startup |
NAck/redelivery + eventual RecoverStale guard |
| Ordering | ORDER BY id (FIFO) |
Stream ordering per subject |
| Daemon.Run() | time.Ticker -> Cycle() -> {poll, claim-loop} |
time.Ticker -> poll() -> publish; sub callback -> process() |
Store.ClaimNext() |
Must remove | No equivalent needed |
Store.EnqueueIssue() |
Optional | Still useful as system-of-record |
| Workflow engine | Unchanged | Unchanged |
| Forge interface | Unchanged | Unchanged |
| events.Bus | Unchanged | Unchanged |