Files
archie/.scratch/explore-daemon-task-loop.md
T
sam bb2193e5e4 feat: add optional NATS JetStream task distribution
Adds optional NATS-based task distribution alongside the existing
SQLite flow. When [nats] is absent from config, zero behaviour change.

New internal/nats package:
- Subject constants matching PRD §2 (archie.task.{bug,feature,default,bootstrap})
- Client: Connect (stream + durable pull consumer), PublishTask
  (Nats-Msg-Id dedup), Fetch, DecodeTask
- Tests with embedded NATS server (connect/publish, dedup, subject routing)

Config: NATSConfig{URL, TokenEnv} on Config, commented-out [nats] block
in config.example.toml

Store: ClaimByIssue atomic claim by owner/repo/number for NATS consumer

Daemon: Nats client field, poll() branches to pollNATS/pollSQLite,
Cycle() branches to drainNATS/drainSQLite (NATS fetch then SQLite
ClaimNext fallback for requeued tasks), processNATSTask
(decode → EnqueueIssue → ClaimByIssue → process → Ack)

Scratch: explorer and plan agent outputs from parallel-explore phase
2026-07-22 02:02:09 +10:00

19 KiB

daemon-task-loop

Here is the full report.


1. Full Lifecycle: GitHub Issue to Completed Task

The lifecycle spans four subsystems: forge (GitHub polling), store (SQLite persistence), daemon (orchestration loop), and workflow (pipeline engine). Here is the exact call chain.

Phase 1 -- Discovery (polling)

Entry: Daemon.Run() at daemon.go:72

  • Creates a time.NewTicker from d.Cfg.PollInterval (default 60s, line 259 of config.go).
  • Calls d.Cycle(ctx) on every tick.

Daemon.Cycle() at daemon.go:88 does three discovery activities, then drains the queue:

  1. d.poll(ctx) -- line 89. Iterates over d.Cfg.Repos, calling d.pollIssues(ctx, repo) (line 140).
  2. d.reconcilePRs(ctx) -- line 90. Finds tasks at StatusPROpen and checks GitHub for merged/closed status.
  3. d.checkWaiting(ctx) -- line 91. Finds tasks at StatusWaitingHuman and LLM-judges human replies.

Daemon.pollIssues() at daemon.go:140 routes by d.Cfg.Dispatch.Trigger:

  • "assignee": calls d.Forge.AssignedIssues(ctx, owner, repo, botUser).
  • "label": calls d.Forge.IssuesWithLabel(ctx, owner, repo, label).
  • "either": calls pollEither() (line 163), which unions both lists deduped by issue number.

Phase 2 -- Enqueue

Daemon.poll() at daemon.go:111-133: for each discovered forge.Issue, calls:

  • d.Store.EnqueueIssue(ctx, owner, repo, number, title, body, labels) -- this is an INSERT ... ON CONFLICT DO NOTHING at store.go:115-118. Returns true if a new row was inserted (idempotency via the UNIQUE(owner, repo, issue_number) constraint at line 99).
  • On insert: posts an emoji reaction via d.Forge.React() (default "eyes", config line 319), sets the "queued" state label, and emits a KindTaskQueued event.
  • On duplicate (inserted==false): calls d.maybeRetryParked() (line 134) which checks whether a human removed the archie:parked label and if so calls d.Store.Requeue().

Phase 3 -- Claim (queue drain)

Back in Daemon.Cycle() at lines 92-106, an inner for loop:

  1. Calls d.Store.ClaimNext(ctx) (line 96).
  2. If nil, returns (queue empty).
  3. Otherwise calls d.process(ctx, task) (line 104), then loops again.

Store.ClaimNext() at store.go:129-141: one atomic SQL statement:

UPDATE tasks SET status='running', attempt=attempt+1, updated_at=datetime('now')
WHERE id = (SELECT id FROM tasks WHERE status='queued' ORDER BY id LIMIT 1)
RETURNING id, owner, repo, issue_number, title, body, labels, status,
    workflow, stage, branch, plan, notes, pr_number, tokens_used,
    iterations, attempt, park_reason

Because SQLite locks the database file for the duration of the UPDATE, no two claimed tasks can overlap -- the daemon is effectively single-consumer. The RETURNING clause returns the full row.

Phase 4 -- Process

Daemon.process() at daemon.go:356:

  1. Looks up the config.Repo for the task via d.repoFor(task) (line 358). Fails/tasks that belong to a now-unconfigured repo go to StatusParked.
  2. Routes to a workflow: wf := workflow.Route(task, d.Workflows) (line 362).
  3. Sets the "working" state label on the issue.
  4. Calls workflow.Run(ctx, wf, taskContext) (line 365).

Phase 5 -- Workflow Execution

workflow.Route() at workflow.go:103-141 decides the workflow:

  1. If t.Workflow is non-empty (set by a prior Requeue), use that exact workflow from the registry.
  2. Otherwise inspect labels: "bug" -> "tdd", "feature" -> "feasibility", "bootstrap" -> "bootstrap".
  3. Falls back to "implement", then "default", then a hardcoded failing error workflow.

workflow.Run() at workflow.go:145-185 executes stages sequentially:

  1. Sets t.Workflow = wf.Name, persists via tc.Store.Update().
  2. For each stage: sets t.Stage, persists, emits KindStageStart, runs stage.Run(ctx, tc), emits KindStageFinish with timing data.
  3. On stage error: if ctx.Err() is set (daemon shutting down), returns cleanly; otherwise calls park() at line 216, which transitions to StatusParked, posts a comment, and emits KindParked.
  4. If tc.Outcome.Status is non-empty after a stage, calls finish() at line 187, which transitions to that terminal status.

Phase 6 -- Terminal States

A workflow terminates through finish() (workflow.go:187) or park() (workflow.go:216):

Outcome Status What happens
park() parked Worktree kept, park-reason comment posted, label set. Human can remove archie:parked to trigger maybeRetryParked().
finish() with StatusPROpen pr_open PR is open on GitHub. Subsequent Cycle() passes call reconcilePRs() which checks merged/closed.
finish() with StatusWaitingHuman waiting_human Daemon watches for human replies via checkWaiting(), LLM-judges approve/reject/unclear.
finish() with StatusMerged merged Final -- PR merged, labels cleared.
finish() with StatusClosedWontDo closed_wont_do Issue closed on GitHub.
finish() with StatusRejected rejected PR closed without merge.

Crash Recovery

Daemon.Startup() at line 52 calls d.Store.RecoverStale(ctx) (store.go:234) which runs:

UPDATE tasks SET status='queued' WHERE status='running'

This re-queues any task that was running when the previous daemon crashed. There is no distributed locking -- the single SQLite file is the authority.


2. Event Struct and Bus

events.Event at events.go:31-42

type Event struct {
    ID       int64          `json:"id,omitempty"` // set by the store sink
    At       time.Time      `json:"at"`
    Kind     string         `json:"kind"`
    TaskID   int64          `json:"task_id,omitempty"`
    Repo     string         `json:"repo,omitempty"`
    Issue    int            `json:"issue,omitempty"`
    Workflow string         `json:"workflow,omitempty"`
    Stage    string         `json:"stage,omitempty"`
    Detail   string         `json:"detail,omitempty"`
    Data     map[string]any `json:"data,omitempty"`
}

Kind constants at events.go:16-26:

  • KindTaskQueued, KindStageStart, KindStageFinish (data: duration_ms, error), KindAgentFinish (data: status, stop_reason, tokens, iterations, model), KindParked (data: reason), KindOutcome (data: status, detail), KindPRMerged, KindPRRejected, KindLog (data: level, msg).

Events are intentionally the single wire type -- every consumer is just another subscriber.

events.Bus at events.go:59-128

A simple in-process fan-out bus:

  • Subscribe(buffer int) *Sub (line 68): creates a buffered channel (default 64), appends to b.subs. Returns a *Sub whose .C field is the read-only receive channel.
  • Publish(e Event) (line 86): stamps At if zero, then iterates all subscribers and does a non-blocking send (select with default). If a subscriber's buffer is full, the event is dropped and s.dropped.Add(1) is incremented. This is the critical design property: backpressure from a slow consumer never blocks the publisher.
  • unsubscribe(sub *Sub) (line 104): removes the sub from the slice, closes its channel.
  • Close() (line 117): sets closed flag, closes all subscriber channels, nils the slice.
  • Bounded per-subscriber buffers with drop-on-overflow, as documented at line 6: "a stalled dashboard connection can never apply backpressure to the task engine."

Publish Side

The daemon has a convenience method emit() at daemon.go:44-49:

func (d *Daemon) emit(e events.Event) {
    if d.Bus != nil { d.Bus.Publish(e) }
}

Called explicitly at every lifecycle transition -- see daemon.go:128 (queued), daemon.go:207 (retried), daemon.go:237 (PR merged), daemon.go:248 (PR rejected), daemon.go:294 (human approved), daemon.go:303 (human rejected).

The workflow engine also publishes via TaskContext.Emit() at workflow.go:61-75, which stamps the task identity and delegates to tc.Bus.Publish().

Subscribe Side (Consumers)

The bus is consumed by at least two subscriber patterns visible in the codebase:

  1. SQLite event log (internal/store/events.go): some goroutine must subscribe and call s.InsertEvent() to persist each event. (The subscription setup is in the composition root cmd/archied. I did not read that file, but InsertEvent at events.go:32 clearly exists for this purpose.)

  2. SSE fan-out for the dashboard: the bus is designed so each HTTP SSE connection gets its own Sub with a small buffer, and the drop-on-overload behavior lets the dashboard fall behind without slowing the daemon.


3. ClaimNext -- SQL, Locking, Concurrency

The SQL

At store.go:129-141:

UPDATE tasks SET status='running', attempt=attempt+1, updated_at=datetime('now')
WHERE id = (SELECT id FROM tasks WHERE status='queued' ORDER BY id LIMIT 1)
RETURNING id, owner, repo, issue_number, title, body, labels, status,
    workflow, stage, branch, plan, notes, pr_number, tokens_used,
    iterations, attempt, park_reason

Three operations in one statement:

  • Subquery: selects the oldest queued task (ORDER BY id LIMIT 1).
  • UPDATE: atomically marks it running, increments attempt.
  • RETURNING: returns the full row, scanned by scanTask() (line 143).

Concurrency Model

  • Single-process, single-threaded consumers. SQLite's locking means only one writer at a time. The daemon has a single Store backed by a *sql.DB with _pragma=journal_mode(WAL)&_pragma=busy_timeout(5000) (line 61). In WAL mode, readers do not block writers, but UPDATE still acquires a write lock -- so two concurrent ClaimNext calls would serialize on the database lock.

  • No distributed claims. There is no advisory lock, no SELECT ... FOR UPDATE, no external coordination. The claim is just the SQL mutation + the WAL busy_timeout (5s). If two daemon processes somehow connected to the same SQLite file, one would get an error or timeout on concurrent UPDATE -- not a correctness issue, but not a supported deployment.

  • Crash recovery, not fencing. Instead of distributed locking, Startup() calls RecoverStale() on boot (store.go:235): UPDATE tasks SET status='queued' WHERE status='running'. This re-queues any task that was in-flight when the previous process died.

  • Single-consumer queue semantics. Daemon.Cycle() at line 92 loops: claim, process, claim, process. There is no parallelism within one daemon instance -- process() is synchronous. The for returns when the queue is empty (ClaimNext returns nil).


4. forge.Issue to store.Task Mapping

forge.Issue at forge.go:10-15

type Issue struct {
    Number int
    Title  string
    Body   string
    Labels []string
}

A pure value type -- no methods, no identity. It represents one GitHub issue at poll time.

store.Task at store.go:30-52

type Task struct {
    ID            int64   // autoincrement PK
    Owner         string  // GitHub owner
    Repo          string  // GitHub repo
    IssueNumber   int     // matches Issue.Number
    Title         string  // matches Issue.Title
    Body          string  // matches Issue.Body
    Labels        string  // comma-separated from Issue.Labels
    Status        string  // lifecycle: queued, running, waiting_human, ...
    Workflow      string  // routed workflow name
    Stage         string  // current stage name
    Branch        string  // worktree branch
    Plan          string  // PRD / implementation plan
    Notes         string  // appended agent notes
    PRNumber      int     // opened PR number
    TokensUsed    int
    Iterations    int
    Attempt       int
    ParkReason    string
    WatchCommentID int64  // human-reply anchor
}

Mapping at Enqueue Time

In Daemon.poll() at daemon.go:112-113:

inserted, err := d.Store.EnqueueIssue(ctx,
    repo.Owner, repo.Name, is.Number, is.Title, is.Body, strings.Join(is.Labels, ","))

The forge-side []string labels are joined into a comma-separated string. The FORGE-side structure is flat (Number, Title, Body, Labels) -- it is the STORE that adds status tracking, routing, and operational fields. The UNIQUE(owner, repo, issue_number) constraint at store.go:99 is the idempotency key: re-polling the same issue does not create a duplicate task.

Mapping at Enqueue SQL

store.go:115-118:

INSERT INTO tasks (owner, repo, issue_number, title, body, labels)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(owner, repo, issue_number) DO NOTHING

Only 6 of 18 Task fields are set at creation. The remaining fields (status defaults to 'queued', workflow/stage/branch etc. default to '' or 0) are filled in during the processing lifecycle.


5. What Changes If ClaimNext Is Replaced with NATS JetStream?

Current Architecture

[Ticker] -> Cycle() -> poll() -> [SQLite EnqueueIssue]
                    -> [SQLite ClaimNext] -> process(task) -> [workflow.Run]

The daemon is a single in-memory loop pulling from SQLite. It is push-based from SQLite's perspective (the daemon polls the DB), but effectively single-consumer -- no parallelism.

Target Architecture with NATS JetStream

[GitHub Webhook] -> [NATS JetStream] -> [Daemon consumes from stream]

or

[Ticker] -> poll() -> [NATS JetStream Publish] 
                    -> [Daemon Subscribe] -> process(task)

What Stays the Same

  1. foreach-over-repos polling (Daemon.poll(), daemon.go:108). The GitHub API polling logic (IssuesWithLabel, AssignedIssues) is independent of the task queue -- you still poll GitHub the same way, you just publish the discovered issue to a NATS subject instead of INSERTing into SQLite.

  2. workflow.Run() and the entire workflow engine (workflow.go:145-185, steps.go, agent.go, feasibility.go, implement.go, tdd.go). The workflow pipeline is agnostic to how tasks arrive -- it receives a *store.Task and a TaskContext. A NATS consumer would still build the same TaskContext and call workflow.Run().

  3. workflow.Route() (workflow.go:103). Label-to-workflow routing remains identical -- it operates on t.Labels and t.Workflow.

  4. forge interface and GitHubClient (forge.go, github.go). The forge is only called to interact with GitHub (comment, label, PR, react) -- not as the task queue. No forge change needed.

  5. events.Bus and all emit() / Emit() calls. The in-process observability bus is orthogonal to task distribution. NATS JetStream would be the task queue, not the event bus; the two are conceptually separate.

  6. Crash recovery via RecoverStale() would be replaced by NATS's exactly-once delivery (with consumer sequence tracking) or at-least-once with deduplication. But the principle is the same: the daemon should be restartable without losing work.

  7. store.Update() and store.Transition() for persisting task progress. Even with NATS as the queue, you still want an audit trail of task state. SQLite is fine for that -- it just becomes the state store rather than the queue.

What Changes

A. Daemon.Run() and Cyc1e() (daemon.go:72-106)

Currently Run() has a ticker that calls Cycle() which does both poll-and-drain. With NATS:

  • The poll ticker would still exist but would Publish() discovered issues to a NATS subject rather than EnqueueIssue.
  • The drain loop would be replaced by a NATS subscription callback. The daemon would register a Pull Consumer or a Push Consumer on the stream, and process() would be called from the message handler, not from a ClaimNext() call.

Something like:

func (d *Daemon) Run(ctx context.Context) error {
    sub, _ := d.JS.PullSubscribe("archie.tasks", ...)
    for {
        msgs, _ := sub.Fetch(1, ...)
        for _, msg := range msgs {
            task := deserialize(msg.Data)
            d.process(ctx, task)
            msg.Ack()
        }
    }
}

B. Store.ClaimNext() is removed (store.go:129).

The atomic SQL UPDATE ... WHERE id = (SELECT ...) is the heart of the single-consumer model. With NATS, the message broker provides the claim -- by delivering the message to exactly one consumer in the consumer group (JetStream's built-in MaxDeliver / Ack / DeliveryGroup). The entire function is replaced by sub.Fetch() or sub.Consume().

C. Store.EnqueueIssue() (store.go:114) becomes optional.

You may or may not persist tasks in SQLite at enqueue time. Options:

  • Keep SQLite as the system of record: the NATS subscriber writes a row on receive, then passes *store.Task to process(). This preserves the dashboard's ability to query task history without replaying the stream.
  • Drop SQLite entirely at the queue level: the NATS message body carries the equivalent of forge.Issue + routing metadata, and the first workflow stage does the persistence.

D. Daemon.Startup() crash recovery changes (daemon.go:52).

RecoverStale() (store.go:234) re-queues tasks left running. With NATS:

  • If a consumer crashes mid-process, the message is either NACKed or redelivered after the AckWait timeout. No explicit recovery step needed.
  • But: if NATS delivers a message and the daemon starts processing (updates task to running in SQLite), then crashes, the NATS message might already be Acked. You still need the crash-recovery query as a safety net, or you redesign to only Ack after the workflow completes.
  • Best practice: Ack after workflow.Run() returns, and keep RecoverStale for the SQLite side.

E. Concurrency model changes fundamentally.

Currently one daemon processes tasks sequentially (one ClaimNext -> process() -> next ClaimNext). With NATS:

  • You can horizontally scale: multiple daemon instances subscribe to the same JetStream consumer group, and NATS round-robins delivery.
  • Within one daemon, you can parallelize: call process() in a goroutine per message, bounded by a semaphore. Each message has an independent worktree, so parallel workflows on different repos are safe. (Current SQLite model could also parallelize, but the synchronous Cycle() design prevents it.)

F. Idempotency moves from SQL UNIQUE constraint to JetStream dedup.

Currently EnqueueIssue uses ON CONFLICT DO NOTHING at store.go:118. With NATS, you would use JetStream message deduplication (Nats-Msg-Id header set to owner/repo/issue_number) to prevent re-processing the same issue enqueue.

G. Task ordering.

Currently FIFO via ORDER BY id LIMIT 1. NATS JetStream supports ordered delivery within a stream, but with multiple repos and priorities this may differ. You would need a stream per repo or use a subject hierarchy like archie.tasks.<owner>.<repo> to preserve ordering semantics.

Summary Table

Component Current (SQLite) NATS JetStream
Task queue tasks table, status='queued' JetStream stream
Claim UPDATE ... WHERE id = (SELECT ...) RETURNING sub.Fetch() / consumer group
Concurrency Single-process, sequential Multi-process, parallel via consumer groups
Idempotency ON CONFLICT DO NOTHING + UNIQUE constraint Nats-Msg-Id dedup
Crash recovery RecoverStale() at startup NAck/redelivery + eventual RecoverStale guard
Ordering ORDER BY id (FIFO) Stream ordering per subject
Daemon.Run() time.Ticker -> Cycle() -> {poll, claim-loop} time.Ticker -> poll() -> publish; sub callback -> process()
Store.ClaimNext() Must remove No equivalent needed
Store.EnqueueIssue() Optional Still useful as system-of-record
Workflow engine Unchanged Unchanged
Forge interface Unchanged Unchanged
events.Bus Unchanged Unchanged