Discovery: split the daemon god object into loop modules #1776

Open
opened 2026-11-02 14:32:56 +00:00 by sam · 4 comments
Owner

From the 2026-08 architecture review (recommendation: strong, top pick).

What is wrong: internal/daemon is one shallow module. *Daemon has 111 methods (85 of them in daemon.go, 2726 lines) and a ~50-field struct mixing the issue poll loop, binding dispatch, NATS task drain, reaction drain, access sweep, and container/storage lifecycle. The interface surface is approximately as large as the implementation: every test constructs the whole struct, and understanding one loop means reading the whole file. 171 commits touched this package in the last 60 days.

Wanted: the poll loop, binding dispatch, and task runner become modules, each behind one narrow interface; Daemon keeps Run/Cycle orchestration only.

Done when:

  • poll, dispatch, and runner each sit behind a single-purpose interface (e.g. Poll(ctx), Dispatch(ctx, CapturedEvent), Submit(taskID))
  • Daemon no longer holds the store/forge/bus fields only those loops need
  • existing daemon tests pass against the new seams; task check green
From the 2026-08 architecture review (recommendation: **strong**, top pick). **What is wrong:** `internal/daemon` is one shallow module. `*Daemon` has 111 methods (85 of them in `daemon.go`, 2726 lines) and a ~50-field struct mixing the issue poll loop, binding dispatch, NATS task drain, reaction drain, access sweep, and container/storage lifecycle. The interface surface is approximately as large as the implementation: every test constructs the whole struct, and understanding one loop means reading the whole file. 171 commits touched this package in the last 60 days. **Wanted:** the poll loop, binding dispatch, and task runner become modules, each behind one narrow interface; `Daemon` keeps `Run`/`Cycle` orchestration only. **Done when:** - poll, dispatch, and runner each sit behind a single-purpose interface (e.g. `Poll(ctx)`, `Dispatch(ctx, CapturedEvent)`, `Submit(taskID)`) - `Daemon` no longer holds the store/forge/bus fields only those loops need - existing daemon tests pass against the new seams; `task check` green
Author
Owner

Refined after review against the platform's actual intake pattern.

Direction (agreed): poll must not stay as methods on *Daemon. The platform's pattern is that intake lives in the source's adapter package and publishes through a narrow func seam — internal/forge/webhook.Receiver does exactly this, wired by the composition root to (*daemon.Daemon).PublishTask, and owns withdrawal via WithdrawFunc. Daemon.poll already no-ops under Forge.Intake == "webhook", so polling is the pull-based fallback.

Contradiction (load-bearing): do NOT add a Poll method to the forge.Forge interface. Forge is a passive CRUD adapter contract (internal/forge/forge.go), core never assumes a forge, and the daemon must run webhook-only. A Poll method would make every forge stateful and leak dispatch/withdrawal policy into the adapter.

Refined plan: extract an issue-source poller module in the forge event family (next to forge/webhook), owning: forge listing calls, workintake.MatchesDispatch, the lastPolled diff and withdrawal bookkeeping, and calls to PublishFunc/WithdrawFunc. Daemon supplies IssueSources, PublishTask, WithdrawTask (already func fields) and keeps Run/Cycle orchestration. Org/identity resolution stays daemon-side via IssueSources; the forge adapter never sees it.

Done when:

  • poller module has one interface (e.g. Poll(ctx) error) and takes sources/publish/withdraw as func deps
  • no poll/withdraw methods remain on *Daemon; Run/Cycle call the poller for Intake == poll|both
  • existing daemon tests pass against the new seam; task check green
Refined after review against the platform's actual intake pattern. **Direction (agreed):** poll must not stay as methods on `*Daemon`. The platform's pattern is that intake lives in the source's adapter package and publishes through a narrow func seam — `internal/forge/webhook.Receiver` does exactly this, wired by the composition root to `(*daemon.Daemon).PublishTask`, and owns withdrawal via `WithdrawFunc`. `Daemon.poll` already no-ops under `Forge.Intake == "webhook"`, so polling is the pull-based fallback. **Contradiction (load-bearing):** do NOT add a `Poll` method to the `forge.Forge` interface. `Forge` is a passive CRUD adapter contract (`internal/forge/forge.go`), core never assumes a forge, and the daemon must run webhook-only. A `Poll` method would make every forge stateful and leak dispatch/withdrawal policy into the adapter. **Refined plan:** extract an issue-source poller module in the forge event family (next to `forge/webhook`), owning: forge listing calls, `workintake.MatchesDispatch`, the `lastPolled` diff and withdrawal bookkeeping, and calls to `PublishFunc`/`WithdrawFunc`. Daemon supplies `IssueSources`, `PublishTask`, `WithdrawTask` (already func fields) and keeps `Run`/`Cycle` orchestration. Org/identity resolution stays daemon-side via `IssueSources`; the forge adapter never sees it. **Done when:** - poller module has one interface (e.g. `Poll(ctx) error`) and takes sources/publish/withdraw as func deps - no poll/withdraw methods remain on `*Daemon`; `Run`/`Cycle` call the poller for `Intake == poll|both` - existing daemon tests pass against the new seam; `task check` green
Author
Owner

Cadence question resolved against the code: polling already runs on the scheduler.

Daemon.BuiltinJobs() (internal/daemon/builtin_jobs.go) lists {ID: "issue-discovery", Name: "Discover forge issues", DefaultInterval: time.Minute, Run: d.poll}, and internal/app/archied/scheduling.go:73-90 seeds these as locked builtin schedules (EnsureBuiltinSchedules) executed by scheduling.NewEngine — so d.poll is invoked by the scheduler, retimable/disableable/runnable-now through the control plane, with the engine's overlap key skipping a second run while one is in flight. Cycle is now only dispatchBindings + drainReactions on the PollInterval intake tick, deliberately not on a schedule: arrived events become tasks there and must not wait on a schedule's coarser cadence (comment in builtin_jobs.go says exactly this). RunOnce runs the builtins directly because one pass has no scheduler.

So #1776's remaining scope is the module split, not the cadence: move the poll implementation out of *Daemon into a forge-family poller module (see previous comment), and the builtin job's Run becomes poller.Poll. The scheduler wiring stays as is.

Cadence question resolved against the code: polling already runs on the scheduler. `Daemon.BuiltinJobs()` (internal/daemon/builtin_jobs.go) lists `{ID: "issue-discovery", Name: "Discover forge issues", DefaultInterval: time.Minute, Run: d.poll}`, and internal/app/archied/scheduling.go:73-90 seeds these as locked builtin schedules (`EnsureBuiltinSchedules`) executed by `scheduling.NewEngine` — so `d.poll` is invoked by the scheduler, retimable/disableable/runnable-now through the control plane, with the engine's overlap key skipping a second run while one is in flight. `Cycle` is now only `dispatchBindings` + `drainReactions` on the PollInterval intake tick, deliberately not on a schedule: arrived events become tasks there and must not wait on a schedule's coarser cadence (comment in builtin_jobs.go says exactly this). `RunOnce` runs the builtins directly because one pass has no scheduler. So #1776's remaining scope is the module split, not the cadence: move the poll implementation out of `*Daemon` into a forge-family poller module (see previous comment), and the builtin job's `Run` becomes `poller.Poll`. The scheduler wiring stays as is.
Author
Owner

Target shape agreed (architecture review + discussion). Recording the plan:

Design: poller as a plugin capability kind.

  • A polling package defines one interface — Poll(ctx) error; pollers emit captures (like schedulecapture.Firer), so binding dispatch turns them into tasks uniformly with webhook/schedule intake.
  • Pull-based sources (the forge, when Forge.Intake == poll|both) register as plugin.Host modules whose manifest declares a "poller" capability kind and implements the interface. Reuses the host's manifest validation, dependency-ordered Start/Stop, and lifecycle — no parallel registry.
  • The polling package keeps a consumer-side list of "poller" modules, the same filtering move the tools provider makes for "tools" (internal/tools/provider/provider.go:114). The host deliberately exposes no domain-callback lookup, so this small addition is in-pattern, not a new bus.
  • Scheduler seeding enumerates registered pollers and seeds one builtin schedule per configured poller: "if it's configured for polling, the scheduler starts a job for it". The generic scheduling.Engine and its overlap key (at-most-one poll in flight per source) stay as they are. Cadence is already scheduler-driven (issue-discovery builtin since fff689312); this change is about who registers what polls, not the cadence.

Explicitly not: a Poll method on the forge.Forge interface (Forge stays a passive CRUD adapter; core never assumes a forge); a second scheduler; per-source schedule rows for dynamic orgs unless a follow-up shows the seed list can track orgs appearing/disappearing.

Done when:

  • Poll(ctx) error exists in a polling package; forge poller emits captures, not direct task publishes
  • builtin schedules are seeded from registered "poller" modules, config-gated
  • no poll/withdraw methods remain on *Daemon; Run/Cycle orchestrate only
  • existing daemon tests pass against the new seams; task check green
Target shape agreed (architecture review + discussion). Recording the plan: **Design: poller as a plugin capability kind.** - A polling package defines one interface — `Poll(ctx) error`; pollers emit captures (like `schedulecapture.Firer`), so binding dispatch turns them into tasks uniformly with webhook/schedule intake. - Pull-based sources (the forge, when `Forge.Intake == poll|both`) register as `plugin.Host` modules whose manifest declares a `"poller"` capability kind and implements the interface. Reuses the host's manifest validation, dependency-ordered Start/Stop, and lifecycle — no parallel registry. - The polling package keeps a consumer-side list of `"poller"` modules, the same filtering move the tools provider makes for `"tools"` (`internal/tools/provider/provider.go:114`). The host deliberately exposes no domain-callback lookup, so this small addition is in-pattern, not a new bus. - Scheduler seeding enumerates registered pollers and seeds one builtin schedule per configured poller: "if it's configured for polling, the scheduler starts a job for it". The generic `scheduling.Engine` and its overlap key (at-most-one poll in flight per source) stay as they are. Cadence is already scheduler-driven (`issue-discovery` builtin since fff689312); this change is about *who registers what polls*, not the cadence. **Explicitly not:** a `Poll` method on the `forge.Forge` interface (Forge stays a passive CRUD adapter; core never assumes a forge); a second scheduler; per-source schedule rows for dynamic orgs unless a follow-up shows the seed list can track orgs appearing/disappearing. **Done when:** - `Poll(ctx) error` exists in a polling package; forge poller emits captures, not direct task publishes - builtin schedules are seeded from registered `"poller"` modules, config-gated - no poll/withdraw methods remain on `*Daemon`; `Run`/`Cycle` orchestrate only - existing daemon tests pass against the new seams; `task check` green
sam added the discovery label 2026-11-02 14:53:55 +00:00
Author
Owner

Correction to my earlier comment: PublishTask is a method (internal/daemon/daemon.go:1345), not a func field; IssueSources and WithdrawTask are the func fields. Forge.Intake is config.Config.Forge.Intake (internal/config/config.go:233), not a forge.Forge field. EnsureBuiltinSchedules is invoked at internal/app/archied/scheduling.go:114.

Correction to my earlier comment: `PublishTask` is a method (internal/daemon/daemon.go:1345), not a func field; `IssueSources` and `WithdrawTask` are the func fields. `Forge.Intake` is `config.Config.Forge.Intake` (internal/config/config.go:233), not a `forge.Forge` field. `EnsureBuiltinSchedules` is invoked at internal/app/archied/scheduling.go:114.
sam added the medium label 2026-11-14 14:45:43 +00:00
sam added the deferred label 2026-11-15 17:26:13 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1776