From d92ff825d9efd7f49a265f973fdf40165e36711a Mon Sep 17 00:00:00 2001 From: Sam Catlow Date: Sat, 5 Sep 2026 02:57:04 +1000 Subject: [PATCH] docs: capture AWX/Ansible inspiration research and service-decomposition PRD Records the AWX pluggable-architecture and Ansible task-model inspiration thread verbatim, the verified current-vs-ideal coupling analysis (webui's in-process HTTP task-action detour, direct gateway internals access, shared config.Holder), the external service-discovery research (K8s-native primary + NATS fallback behind one ServiceRegistry seam), and a Proposed-status PRD recording the target service shape and open questions before any implementation work is filed. --- ...ce-decomposition-and-ansible-task-model.md | 143 +++++++++ ...e-reference-awx-decomposition-vs-archie.md | 126 ++++++++ .../service-discovery-research-2026-09-05.md | 298 ++++++++++++++++++ docs/prds/service-decomposition.md | 147 +++++++++ 4 files changed, 714 insertions(+) create mode 100644 docs/inspiration/2026-09-05-awx-service-decomposition-and-ansible-task-model.md create mode 100644 docs/inspiration/module-reference-awx-decomposition-vs-archie.md create mode 100644 docs/inspiration/service-discovery-research-2026-09-05.md create mode 100644 docs/prds/service-decomposition.md diff --git a/docs/inspiration/2026-09-05-awx-service-decomposition-and-ansible-task-model.md b/docs/inspiration/2026-09-05-awx-service-decomposition-and-ansible-task-model.md new file mode 100644 index 000000000..5644374a1 --- /dev/null +++ b/docs/inspiration/2026-09-05-awx-service-decomposition-and-ansible-task-model.md @@ -0,0 +1,143 @@ +# Sam's message, verbatim (2026-09-05) + +Source: chat message pasting an AWX forum post (AWX modernization series, +"Refactoring AWX into a Pluggable, Service-Oriented Architecture") plus a +terminal transcript and a follow-up message including Ansible's +`ansible/playbook/task.py`. Preserved verbatim below for later reference. +Image attachment (`AAP Pluggable Architecture` diagram) is described, not +reproduced, since this file is text-only. + +--- + +[01:09:53] sam@helix /work/clones +> git clone https://github.com/ansible/ansible.git +Cloning into 'ansible'... +remote: Enumerating objects: 644471, done. +remote: Counting objects: 100% (21/21), done. +remote: Compressing objects: 100% (21/21), done. +remote: Total 644471 (delta 5), reused 2 (delta 0), pack-reused 644450 (from 3) +Receiving objects: 100% (644471/644471), 251.86 MiB | 9.60 MiB/s, done. +Resolving deltas: 100% (427812/427812), done. +[01:12:50] sam@helix /work/clones +> git clone https://github.com/ansible/awx.git +Cloning into 'awx'... +remote: Enumerating objects: 331428, done. +remote: Counting objects: 100% (399/399), done. +remote: Compressing objects: 100% (256/256), done. +remote: Total 331428 (delta 259), reused 153 (delta 143), pack-reused 331029 (from 3) +Receiving objects: 100% (331428/331428), 346.81 MiB | 12.94 MiB/s, done. +Resolving deltas: 100% (254095/254095), done. +[01:14:59] sam@helix /work/clones +--- +[Image #2] + +Hi folks, + +In our previous post, we discussed the upcoming changes we're making to the AWX project. Today, we'd like to dive deeper into one of the most significant changes: refactoring AWX into a pluggable, service-oriented architecture. +Current State of AWX + +AWX is currently structured as a monolithic Django application. It consists of several entry points, each running as individual processes, including our API, Job Dispatcher, Event Processor, and Websocket subsystem. Alongside these, AWX relies on various ancillary processes such as Redis, Rsyslog, Receptor, Nginx, Uwsgi, and Daphne. +Challenges with the Current Architecture + +Over time, it has become increasingly challenging to implement changes due to the large, self-contained nature of the application. This monolithic structure has not only slowed down our development process but also made it difficult to ensure efficient code reuse across our other applications. Consequently, this impacts both our development efficiency and the ability to quickly iterate on features and fixes. As we continue to extend capabilities across Ansible Automation Platform we cannot afford these challenges and maintain upstream projects the way they are currently structured. +Proposed Changes + +To address these challenges, we are beginning to transition AWX towards a more pluggable, service-oriented architecture. By breaking down the monolithic structure into smaller, modular services, we aim to enhance both flexibility and maintainability. This shift will enable us to make changes more swiftly and efficiently in AWX while also improving our ability to reuse code across various projects leveraged by Ansible Automation Platform. +Implementation Plan + +Our implementation plan involves several key steps: + + Identifying Modular Services: We will start by identifying the core functionalities that can be extracted into individual plugins and/or services. + Creating a Communication Framework: Establish a robust communication framework to ensure seamless interaction between these services. + Gradual Transition: Implement the new architecture in phases to allow for continuous feedback and improvement. + Community Collaboration: Engage with our community throughout the process to gather feedback and ensure the new architecture meets the needs of our users. + +Future-Looking Architecture + +1316×628 38.8 KB + +Expected Outcomes + +By refactoring AWX into a service-oriented architecture, we expect several positive outcomes: + + Improved Flexibility: Easier to implement changes and add new features. + Enhanced Maintainability: Simplified codebase with clear service boundaries. + Better Code Reuse: More efficient reuse of code across different projects. + Active Community Participation: Greater opportunity for community members to contribute and shape the project. + +Call to Action + +We invite feedback and collaboration with us on this significant transformation by replying to this forum post. + +Join the discussions on our forum, share your thoughts, and help shape the future of awx . +Conclusion + +These architectural changes are a crucial step towards a more modern and scalable AWX. We look forward to your participation and support as we embark on this journey together. + +Stay updated by joining The Forum, following the +News & Announcements and +awx tags. + +Thank you for your continued support. +Links in this AWX update series + + Blog: Upcoming Changes to the AWX Project + Streamlining AWX Releases + Refactoring AWX into a Pluggable, Service-Oriented Architecture (this post) + Upcoming changes to AWX Operator installation methods + AWX UI and credential types transitioning to the new pluggable architecture + AWX modernization: Moving forward + AWX modernization: Ansible UI + AWX modernization: Ansible Jewel + +Useful links + + 2024-05-30 Blog: Upcoming Changes to the AWX Project + 2024-07-01 Streamlining AWX Releases + The Forum: AWX topics + The Forum: Newsletter Category +--- +Taking a look at this, How could we reframe ourselves to capture some of this market? + +Archie is already a beast of a project, I think that it would be an excellent idea to think about cleaning up, simplifying, and unifying many of it's layers down to cleaner interfaces and even if it's starting at the most basic layers such as splitting the UI from the backend, turning the agent into a core runner, or even having 2 separate agent definitions, an AI agent and an agent runner. or. even better, an agent runner which utilises an open source project like Pi as the AI harness instead of having to code our own one. + +I have Pi running now, with an immensely fast and capable model (deepseek-v4-flash-vision-exp) which is absolutely able to perform many actions at great speed, and that could assist us to getting this chunk of work done. + +the idea of everything being pluggable services - making the entire solution modular, is exactly the goal I've had from the start. and it's already heading that way, but it's missing some guidance, assistance and care. I believe that a multi-agent system could help me get there, where I talk to an agent like claude, which spawns subagents to start implementation in isolation, and another agent monitors the work, assists, and carefully merges as the migration is in progress. + +[01:23:10] sam@helix /work/apps/archie-core +> tree +(directory tree of archie-core omitted here -- see repository working tree at +the time of writing for the full listing; top-level shape was: AGENTS.md, +ARCHITECTURE.md, cmd/{archie-agent,archied,archie-playbooks}, deployments/, +docs/{architecture,archive,data,guides,prds,public}, examples/, extras/, +internal/{agentexec,app,channels,config,container,daemon,domain,eventbus, +events,forge,forgerpc,gate,gateway,indexing,infrastructure,installtype, +logging,memory,natsrpc,pairing,plugin,ratelimit,releaseannounce, +releaseupdate,secret,skill,skillscript,storage,store,storerpc,taskrun, +taskstate,tools,webhookguard,webui,worktree,worktreerpc,yaegiutil}, scripts/, +tools/, ui/{dist,src,test}) +[01:23:12] sam@helix /work/apps/archie-core + +--- + +One other thing I've grabbed inspiration from the Ansible project and I'm hoping to splice in - is the task.py (and likely more items) definitions; I think that being able to use existing ansible playbooks (and eventually roles/etc) within archie, would be astoundingly benefitial for engagement. + +(ansible/playbook/task.py, GPLv3, Michael DeHaan et al. -- full file pasted in +chat; omitted here as reference-only source code, not reproduced verbatim in +this repository. It defines Ansible's `Task` class: FieldAttribute-based +declarative fields such as `args`, `action`, `loop`, `register`, +`changed_when`/`failed_when`/`until`, `delay`/`poll`/`retries`, +`loop_control`; `preprocess_data` which runs `ModuleArgsParser` to resolve the +legacy free-form task shape into `(action, args, delegate_to)`; `post_validate` +which templates fields through Jinja2 (`TemplateEngine`); parent-chain +attribute inheritance (`_get_parent_attribute` walking up through Block / +TaskInclude / Play); and `_resolve_conditional` for `when:` evaluation.) + +--- + +Attached image: "AAP Pluggable Architecture" diagram -- a decomposition tree +with "AWX Decomposition" as root, branching to: Credentials, Subscription/ +Billing, Auth (children: RBAC, SSO, Policy/ABAC), Communication (child: +Inter Service, and Execution (mesh) linked back into Runtime), Telemetry, +Inventory, Runtime (child: Scheduler), Content (child: Projects). diff --git a/docs/inspiration/module-reference-awx-decomposition-vs-archie.md b/docs/inspiration/module-reference-awx-decomposition-vs-archie.md new file mode 100644 index 000000000..9c66f6bb4 --- /dev/null +++ b/docs/inspiration/module-reference-awx-decomposition-vs-archie.md @@ -0,0 +1,126 @@ +# Module reference: AWX decomposition / Ansible task model vs. Archie + +Status: exploratory reference, not a design decision. Nothing here is +authoritative -- `docs/architecture/` stays the source of truth. This exists +so a later PRD (if one gets written) doesn't have to re-derive "what do we +already have vs. what AWX/Ansible are describing." See +[[2026-09-05-awx-service-decomposition-and-ansible-task-model]] for the +source material this is built from. + +Two follow-on artifacts extend this line of exploration: +- Two Archify diagrams contrasting current vs. an ideal service-oriented + shape: `/work/artefacts/archie-core-current-state-archify.html` and + `/work/artefacts/archie-core-ideal-state-archify.html`. The current-state + diagram is grounded in verified coupling (webui's task-action path routes + through its own HTTP handler in-process; direct `gateway.SessionStore` / + `Router` / `TurnHistory` access; a shared `config.Holder`) -- not the + "already split" claim this document's earlier draft mistakenly made. +- [[service-discovery-research-2026-09-05]] -- externally researched + comparison of service-discovery mechanisms for the ideal-state's "install + a service, the fleet auto-aligns" requirement. Recommends Kubernetes-native + discovery as primary (Helm chart + Operator on k3s / a real cluster) with + NATS-based discovery (already a hard dependency) as the built-in fallback + for no-K8s single-host installs, both behind one `ServiceRegistry` + interface. + +## Reading the two source ideas separately + +The AWX post and the Ansible `Task` paste are two different asks and should +stay two different tracks: + +1. **AWX's decomposition tree** is an argument for splitting a monolith + along service boundaries (Auth, Comms, Runtime, Content, ...) with a + pluggable communication mesh between them. Archie's six-domain split + (`docs/architecture/index.md`) already **is** that argument, applied + earlier and at a different grain (domain boundaries in one Go module + + NATS transport, not separate deployable services). The gap isn't "do we + need boundaries," it's "are the boundaries clean and are they enforced." +2. **Ansible's `Task`/module system** is a request for a *content* format: + let users write YAML tasks against a stable action vocabulary and have + Archie interpret them, the way Ansible interprets `- name: ... action: + module: args`. Archie has the skeleton of this already (EDA playbook + engine, Module action position) but it's deliberately narrow today. + +## AWX decomposition tree -> Archie's existing domains + +| AWX node | Archie equivalent today | Location | +|---|---|---| +| Auth (RBAC / SSO / Policy-ABAC) | `internal/domain/binding` (task-to-identity/config binding) + `internal/policy` (cross-cutting, zero internal imports) + forge/channel auth (Telegram pairing, webhook HMAC) | `internal/domain/binding/`, `internal/policy/`, `internal/pairing/`, `internal/webhookguard/` | +| Communication / Inter-Service | NATS (`ARCHIE_TASKS` JetStream work queue, core-NATS RPC via `_INBOX.*`) | `internal/infrastructure/eventbus/nats/`, `internal/natsrpc/`, `internal/eventbus/` (contract, zero internal imports) | +| Execution (mesh) | Container pool running isolated agent worktrees over NATS, not a mesh of remote execution nodes (Ansible's Receptor equivalent doesn't exist -- Archie's "remote execution" is a Docker container, not a fleet) | `internal/container/pool.go`, `internal/agentexec/`, `internal/worktree/` | +| Runtime / Scheduler | `internal/domain/scheduling` (ticker contract) + cron-driven work intake | `internal/domain/scheduling/`, `internal/infrastructure/cronstore/` | +| Content / Projects | Nearest analogue is the **EDA playbook + module system** (below) plus `internal/skill` (skill catalog) -- Archie has no "project" = git-checkout-of-automation-content concept the way AWX does | `internal/domain/eda/playbook/`, `internal/domain/eda/module/`, `internal/skill/` | +| Telemetry | `internal/logging` (task sink, feed, registry) + `internal/events` | `internal/logging/`, `internal/events/` | +| Inventory | No direct equivalent. Closest is forge-side repo/identity config (`[[identities]]`, `internal/forge`) -- Archie targets one identity acting across a small set of forge repos, not a dynamic inventory of managed hosts | `internal/forge/`, `internal/config/` | +| Credentials | `internal/secret` (engine host, env/age/sops/vault engines) | `internal/secret/`, `internal/secret/enginehost/` | +| Subscription/Billing | No equivalent -- Archie is self-hosted, single-tenant; this AWX node doesn't map to anything and shouldn't | -- | +| UI split from backend | Already true: `internal/webui` (Go API server) serves a separately built SPA from `ui/dist` (Vite/React-ish, `ui/src//`) over HTTP/SSE. There is no shared-process coupling to undo. | `internal/webui/`, `ui/src/` | + +**Read on "reframe to capture market":** the honest gap next to AWX/AAP isn't +domain boundaries -- Archie already has those, arguably cleaner (env-enforced +gates, one dependency direction, contracts with zero internal imports). The +gap is (a) no content/inventory layer letting non-programmers hand Archie +declarative automation the way an Ansible playbook does, and (b) everything +still ships as one Go binary + one container image, not independently +deployable/upgradable services. (a) is the more differentiated, more +buildable opportunity given a one-person team; (b) is expensive AWX-style +service splitting with limited payoff at Archie's current scale and should +not be pursued for its own sake. + +## Ansible `Task` model -> Archie's EDA playbook/module system + +Ansible's `task.py` is the runtime representation of one YAML task after +`ModuleArgsParser` resolves the legacy free-form shape and `TemplateEngine` +renders Jinja2. Archie's nearest shipped analogue is the EDA playbook engine, +which is **intentionally much narrower right now**: + +| Ansible concept | Archie's current shape | Gap / what porting the idea would mean | +|---|---|---| +| `Task` (one step, `action` + `args`) | `playbook.Action` (`internal/domain/eda/playbook/playbook.go`) -- but **exactly one action per playbook, position must be `"workflow"`**, hard-rejected at load otherwise | Multi-action, multi-position playbooks are the literal ask ("use existing ansible playbooks... within archie") but are explicitly blocked today on unresolved execution-time gaps: mid-run failure semantics, idempotency for non-workflow actions (see the package doc comment and `docs/architecture/plugins-and-extensions.md`'s Event sources section). This is a known, named boundary, not an oversight. | +| Module (`ansible.builtin.*`, arbitrary Python) | `module.Kind` registry, Yaegi-interpreted `.go` files against a generated per-kind symbol table (`internal/domain/eda/module/`) -- **one shipped kind: `log`** | Ansible modules are dynamically loaded from a huge ecosystem; Archie's Module position is closed-registry-of-named-kinds, each with its own hand-written contract package (`internal/domain/eda/module/`). Adding "run a shell command," "call a forge API," "send a channel message" module kinds is possible under the existing pattern but each is new work, not automatic from having the mechanism. | +| `when:` conditional | `Action.When`, compiled CEL (`internal/domain/eda/expr`), not Jinja2 | Already decided: CEL over Jinja2 (open question 1 in the playbook engine PRD, resolved). Don't reopen. | +| `register` / fact accumulation across tasks | Not present -- `DispatchInput.Event` is a flat map exposed to CEL, there's no cross-action result chaining because there's only one action | Needed before multi-action playbooks are meaningful; tracked implicitly by the same "unresolved execution-time gaps" blocker above. | +| `loop` / `until`/`retries`/`delay` | Not present | Same -- follows from allowing more than one action per playbook. | +| Field-attribute inheritance (task -> block -> play) | No block/parent chain -- a playbook is flat: one `Trigger` + one `Action` | Ansible's play/block/role hierarchy has no Archie counterpart yet; would only matter once multi-action/roles are in scope. | +| Roles (reusable task bundles) | Nearest is `internal/skill` (skill catalog, frontmatter, tool defs) -- conceptually closer to Claude Skills than Ansible roles | Not a role system; skills are agent-facing capability bundles, not declarative task sequences. | +| Channel / Forge action positions | Named and designed but explicitly **not implemented** (`docs/architecture/plugins-and-extensions.md`: "Channel and Forge action positions are designed but not yet implemented," open investigation `archie-core-t2db.19`) | This is the direct next step toward "playbooks that do things," independent of the Ansible-compatibility idea. | + +**What porting real Ansible playbooks would actually require**, in rough +dependency order, if this becomes a real proposal later: + +1. Lift the playbook engine's hard boundary: multi-action playbooks, which + needs the mid-run failure/idempotency semantics the doc comment flags as + the blocker today. +2. A result-passing mechanism between actions (Ansible's `register`) -- + currently nothing carries one action's output into the next action's + `when`/`args`. +3. Loop/retry primitives per action. +4. A much larger Module `Kind` registry, or a deliberate decision to accept + a narrower "Archie automation language inspired by Ansible's shape" rather + than "runs unmodified Ansible YAML" -- the latter also drags in Jinja2, + the module_utils Python runtime, and inventory/host-targeting concepts + Archie has no use for (Archie targets git repos and forge issues, not + fleets of managed hosts). Recommend treating "Ansible-*inspired* syntax, + not Ansible-*compatible* execution" as the working assumption if/when a + PRD gets written -- it avoids importing Ansible's inventory/host model, + which doesn't fit Archie's single-repo-per-task execution shape at all. + +## On the multi-agent / Pi-harness-as-runner idea + +Out of scope for this reference doc (it's an operating-model question, not a +module-mapping one) but noted since it was raised alongside the above: this +is a `migration-decisions.md`-track question (agent execution as a strict +data boundary is a named invariant in `ARCHITECTURE.md`), not something to +back into via the EDA/module work. Flag separately if it's worth pursuing. + +## Bottom line + +- No code changes implied by this document. It's a map, not a proposal. +- The two ideas in Sam's message decompose cleanly: AWX's service-split + argument is mostly already satisfied by Archie's domain architecture; the + Ansible-playbook-content idea is a real, currently-blocked extension of + the EDA Module/Channel/Forge action positions, gated on the same + execution-semantics work already named in `plugins-and-extensions.md`. +- If this becomes a real initiative, it belongs in `docs/prds/` as a + decisive 1-pager per `AGENTS.md`'s scope-discipline rule -- not as an + elaboration of this file. diff --git a/docs/inspiration/service-discovery-research-2026-09-05.md b/docs/inspiration/service-discovery-research-2026-09-05.md new file mode 100644 index 000000000..e496b4f45 --- /dev/null +++ b/docs/inspiration/service-discovery-research-2026-09-05.md @@ -0,0 +1,298 @@ +# Service discovery research for the archie-core service decomposition + +Status: exploratory research, not a settled decision. This backs the "ideal +state" service-oriented sketch in +[[module-reference-awx-decomposition-vs-archie]] and its two Archify +diagrams (`/work/artefacts/archie-core-current-state-archify.html`, +`/work/artefacts/archie-core-ideal-state-archify.html`). Nothing here commits +the codebase to anything; if the service decomposition itself gets approved, +this is the input to the actual `docs/architecture/migration-decisions.md` +entry and/or a `docs/prds/` design doc, not a substitute for either. + +Produced by an external research pass (Pi subagents plus supervisor-run +`deep_search_exa` queries against primary sources: NATS discovery, K8s-native +discovery, Consul operational footprint, gRPC/xDS/mesh weight). The K8s lane +detached mid-flight and was recovered via supervisor reply, covered from +fetched Kubernetes docs (Service/EndpointSlice/CoreDNS/Operator + k3s Helm). +Reproduced here close to verbatim as the record of that research; light +formatting only. + +Business framing this research was scoped against, per Sam (2026-09-05): the +decomposition is being evaluated as a real product distribution strategy — +possibly sold self-hosted, possibly as a managed/hosted offering (the +RHEL/Ansible Automation Platform model: same product, sold both ways), not +ruled to either shape. Evaluate infrastructure choices on technical merit for +that audience, not through the lens of Sam's own homelab deployment +preferences (see `feedback_dont_project_homelab_prefs_onto_product` in +Claude's project memory). "Cost" in this document means operational +complexity, never dollar price — pricing and SaaS-plan questions are +explicitly out of scope here. + +--- + +## 1. Candidate mechanisms + +| # | Mechanism | Who stands it up | Health-aware endpoints? | "Install → auto-join" mechanics | Single-host fit | Multi-host fit | +|---|---|---|---|---|---|---| +| 1 | **Kubernetes-native** (Service + EndpointSlice + CoreDNS), delivered via Helm chart + Operator, on k3s (single-node) or a real cluster | A whole (even if small) control plane: api-server, etcd, scheduler, controller-manager, CoreDNS, kube-proxy. k3s packages this into a ~one-command install | Yes, free — readiness probes gate which pods land in an EndpointSlice | Deploy/helm install a Service (or an Operator reconciles a CR) → CoreDNS serves the name → existing services dial it | Strong (k3s is ~½ GB RAM) | Good (real cluster, or k3s server+agents) | +| 2 | **NATS-based discovery** (micro `$SRV` announce + JetStream KV registry) | NATS broker — already a hard dependency, so nothing new to stand up | No — only connection liveness; you write heartbeat+TTL for KV | Service connects to the bus on start, publishes/announces → peers observe the join | Excellent | Workable but requires a real NATS cluster / leaf nodes (TLS/NKey/JWT weight) | +| 3 | **Consul** (registry + agent + DNS) | A second control plane: 3–5 Raft servers + a client agent on every node | Yes (TCP/HTTP/gRPC/TTL) | Agent registers with server; peers resolve via Consul DNS | OK (one server + agents) | Good (multi-DC, WAN federation) but per-node agent burden | +| 4 | **etcd** (discovery substrate) | A Raft KV cluster you operate | Not natively — you own catalog/health semantics | Lease registration + prefix watch | OK (embeddable) | Good (R3 replication) but you build the semantics | +| 5 | **gRPC xDS / service mesh** (Istio/Linkerd/Traffic Director, proxyless gRPC) | A control plane (+ per-pod sidecar if mesh) | Yes (xDS health) | Register with control plane → EDS pushed to subscribed clients | Heavy / overkill | Heavy; K8s-first; sidecar cost (~0.2 vCPU + 60 MB/pod @1k rps in Istio-sidecar-land) | +| 6 | **DNS-SRV / mDNS** | Nothing new | No native health; TTL caching breaks fast failover | Register a record | mDNS yes (but link-local); DNS-SRV needs a resolver you control | mDNS fails at the first router/VLAN; DNS-SRV needs your own resolver + a custom gRPC resolver (grpc-go doesn't consume SRV natively) | + +## 2. What this narrows to + +Six candidates surfaced, but the field collapses to two real finalists once +three filters specific to archie-core apply: (a) the fleet is ~7 Go +services, (b) a gRPC-sync + NATS-async mix is already the working +assumption, and (c) the defining requirement is "customer installs a +service, the fleet auto-aligns, no redeploy/reconfig." + +- **Consul / etcd are dominated by K8s-native.** Consul adds a whole second + control plane (Raft server quorum + a client agent on every node you don't + control) — redundant if the fleet lives in K8s (buying a second control + plane to replace one you already have), heavy if it doesn't. etcd makes + you own catalog + health semantics, which is exactly what K8s-native and + NATS each give for free in different ways. Both remain viable, just not + better. +- **gRPC xDS / service mesh is the wrong tool at this scale.** Field + consensus (multiple 2026 sources): don't adopt a mesh until 20+ services; + archie-core is at ~7, single-language (Go), with resilience logic already + in Go. xDS/proxyless-gRPC earns its keep only with locality-aware routing + / canary / circuit-breaking across hosts, which nothing here signals. The + durable takeaway isn't "use xDS"; it's that gRPC is resolver plumbing, not + a registry — whatever registry gets picked feeds gRPC through a resolver + (DNS resolver, or a custom one). +- **DNS-SRV / mDNS are insufficient.** mDNS is link-local (dies at + router/VLAN/container boundaries), so it can't span routed multi-host; + DNS-SRV has no health and grpc-go's default resolver doesn't consume SRV + natively anyway. + +So the genuine decision is K8s-native vs. reuse-NATS. + +## 3. K8s-native vs. NATS: an honest weigh + +**Where K8s-native wins decisively:** +- The only mechanism that delivers "install a service → the fleet + auto-aligns" at the infrastructure layer, with health-filtered endpoints, + where the customer's install is a straightforward "apply a Deployment + + Service" (or a Helm chart, or an Operator reconciling a CR). This is the + single hardest, most product-defining requirement, and K8s solves it + natively. +- Delivers, as a side effect, almost everything a 7-service fleet needs + anyway: health (readiness/liveness probes), orchestration + (restart/scale), config (ConfigMaps/Secrets), rollouts, clean external + exposure (Ingress/Gateway). Not extra weight for its own sake — buying + pieces that would otherwise be hand-built. +- The hosted/managed path is free. If the product is ever run for customers + (the "sold both ways" shape kept open above), K8s runs anyway. + K8s-native discovery makes self-hosted and hosted deployments use + identical discovery semantics — one codebase, no divergence. This is the + RHEL/Ansible Automation Platform lesson: same product, sold both ways, + same internal architecture. + +**Where NATS wins:** +- Zero new infrastructure. NATS is already a hard dependency and already in + the deployment (embedded, or the existing `docker-nats-stack.toml`). For + the lowest-end single-host customer who wants "one binary dropped on a + box," NATS discovery means no cluster, no K8s — just the bus already + running. +- Genuinely idiomatic: a service connects, announces via the micro `$SRV` + tree (PING/INFO/STATS) or a JetStream KV record, and disappears the + moment its connection closes. Re-join = "open the connection and + publish." No per-service config, no sidecar, no hand registration. + +**Where NATS genuinely falls short for this product:** +1. It's a NATS service locator, not a gRPC endpoint registry. The `$SRV` + framework and the KV bucket hand you names/responders, not `host:port` + gRPC endpoints. Each service's `host:port` has to be carried in + metadata/KV and a custom gRPC resolver hand-written; gRPC's built-in + resolver/health model won't "just work." Real integration work owned in + the sync path — now the product's primary contract. +2. No native health. NATS has no last-will or probe. Heartbeat+TTL (KV) + must be implemented to distinguish "installed but unhealthy" from "not + installed," and stale entries are your problem. K8s readiness probes do + this for free. +3. Discovery is coupled to broker availability. If NATS is down, every + service looks down (a hard SPOF unless NATS is itself clustered). + Multi-host is where the cost shows: a real NATS cluster or leaf nodes, + with TLS/NKey/JWT credential management — operator weight, not "just + works." + +The balance: NATS is a legitimate fallback and a genuine part of the +answer; it is not a substitute for the registry that resolves gRPC +endpoints with health. If the product's sync path is gRPC, the registry +serving it should be K8s-native (DNS → resolver, headless-Service SRV for +ports, EndpointSlice health) rather than a hand-built NATS registry. + +## 4. Recommendation + +**Primary:** Kubernetes-native service discovery (Service + +headless-Service/DNS + EndpointSlice, served by CoreDNS), delivered as a +Helm chart + Operator, on k3s for single-host customers and a real cluster +for multi-host / hosted. + +**Fallback:** NATS-based discovery (micro `$SRV` announce + JetStream KV +registry) for no-K8s single-host installs, using a custom gRPC resolver +that reads `host:port` from the KV bucket. + +**Rationale, short version:** +- K8s-native is the only candidate that solves the hardest requirement — + "install a service and the rest of the system joins in to align" — at + the infrastructure layer with health-filtered endpoints and a + customer-comprehensible install. Not extra weight for its own sake; it + replaces a stack of pieces a 7-service product needs anyway, and makes + the multi-host + hosted paths free from one codebase. +- k3s packages the control plane into a ~one-command install a customer can + stand up, and the Operator/Helm/CRD pattern is exactly the "install + Curator, Gateway picks it up" product story. +- NATS is not discarded — it stays in the fleet for async fan-out, and + becomes the second discovery implementation so the smallest no-K8s + customer keeps "one binary on a box." The fallback is the honest + acknowledgment that a full K8s cluster is genuinely heavy for the very + smallest single-node customer. + +Why not NATS-only: it's a service locator needing hand-built gRPC endpoint +resolution + health, and it couples discovery to broker availability. Why +not Consul/etcd: a redundant second control plane (if K8s) or "you own the +semantics" (etcd), with per-node agents. Why not a mesh: overkill at ~7 Go +services; consensus is 20+. + +### The seam that keeps both shapes on one codebase + +The Gateway (and any service that routes to peers) should talk to discovery +through one internal interface — + +```go +type ServiceRegistry interface { + Resolve(service string) []Endpoint + Watch(service string) <-chan Event // join/leave +} +``` + +— with two implementations: a K8s one (DNS / EndpointSlice / custom +resolver) and a NATS one (KV / `$SRV`). The rest of the code is +registry-agnostic. This is what makes "K8s-native primary, NATS fallback" a +coherent architecture rather than a fork: business logic doesn't change, +only the registry injection point does. + +## 5. End-to-end: "install Curator, Gateway discovers it" + +**Path A — K8s-native (primary):** +1. Customer installs the product (`helm install archie`, or pre-provisioned + on k3s). Base services (Gateway, UI, Messaging, Scheduler, Execution, + State Store) each deploy as a Deployment + Service; Gateway is always up + and already holds a resolver for every known service name. +2. Customer installs Curator: `helm install archie-curator`, or the + Operator watches a `Curator` Custom Resource and reconciles it — creating + a Deployment (pods) + a `curator` Service. +3. The Service's label selector matches the Curator pods; the endpoint + controller writes their IPs into an EndpointSlice. Readiness probes gate + which pods are "ready" (a Curator that only registers once healthy won't + receive traffic). +4. CoreDNS immediately serves `curator..svc.cluster.local` (A record; a + headless Service additionally gives an SRV record with the port). +5. The Gateway resolves `curator..svc.cluster.local` through its gRPC + DNS resolver (or a K8s-aware resolver). On the next re-resolution/update + it picks up the new endpoint. No Gateway redeploy, no Gateway config + change — and because it's DNS, the Gateway code doesn't even need to know + it's running under K8s. +6. Removal: `helm uninstall archie-curator` (or scale to 0 / delete the CR) + → EndpointSlice empties → DNS record disappears → Gateway's resolver + drops it. Clean. + +**Path B — NATS fallback (single-host, no K8s):** +1. One NATS (embedded, or external via the existing compose stack) is the + shared bus; its URL is part of the base install. +2. Every service, on start, connects to that URL and either (a) registers + via the micro framework (`AddService("curator", "1.0.0", …)`), appearing + on the `$SRV` tree, or (b) writes its `host:port` + version into a + JetStream KV bucket (`archie/registry/curator/`) and heartbeats it + with a TTL. +3. The Gateway holds a live roster (from `$SRV.INFO` or a KV watch) and + feeds it into a custom gRPC resolver mapping service-name → `host:port`. +4. Customer starts one Curator binary (`archie-curator`, or `docker run` + the image) with the same NATS URL in its config. It connects, announces, + and heartbeats. The Gateway's roster observes the new entry (within one + announce/heartbeat window) and starts routing to it. No Gateway restart, + no config change beyond the shared NATS URL already present in the base + install. +5. Removal: stop Curator → its connection closes → it leaves `$SRV` + immediately and its KV record expires via TTL. The Gateway routes that + capability to "not installed." + +## 6. Failure-mode honesty + +- **K8s-native:** pod death → readiness fails → dropped from EndpointSlice + within seconds, so the gateway stops routing to it (health-aware). DNS + TTL caching is mostly harmless because a Service's ClusterIP is stable + and all churn lives behind it in the EndpointSlice. If the control plane + goes down, already-running pods keep serving and DNS still resolves from + cache; only new scheduling/propagation halts. Known gap: cross-cluster + and off-cluster consumers (client-side resolvers, a worker fleet on VMs) + need ExternalDNS or a bridge. mTLS/auth, if needed, is a separate concern + (NetworkPolicy / cert-manager), not the mesh. +- **NATS fallback:** service dies → dropped from `$SRV` immediately, KV + record lingers until heartbeat-TTL (bounded staleness). Broker + unreachable → every service appears down (hard SPOF unless NATS is + clustered). Plain subjects aren't a consensus store (split-brain across + partitions possible); JetStream KV gives Raft per stream but KV placement + across hosts needs its own decisions. Health semantics must be built by + hand. + +Both paths are robust against "a service dies"; the difference is who does +the health checking — K8s readiness probes do it for free, NATS makes you +build it. + +## 7. Addendum — recovered K8s lane, optional-service semantics + +A separately recovered K8s-focused research lane landed independently on +the same verdict (K8s-native as the canonical, declarative, self-healing +fleet-discovery answer; fits multi-host and the AAP-style "sold both ways" +model best) and confirmed the same install → Operator → CoreDNS → resolver +flow above. It sharpened one point specific to archie-core's shape: + +**The optional, intermittently-running Curator needs explicit "absent" +semantics under K8s.** A scaled-to-zero or not-yet-created Curator has no +ready endpoint and no DNS record — the Gateway's gRPC `dns://` resolver +gets `NXDOMAIN`/empty resolution, and this must be treated as "Curator is +disabled," not "Curator is broken." Concretely: +- Lazy, on-demand re-resolution — don't pin a dead subchannel as a hard + failure. +- A distinct `NotInstalled` vs. `Down`/`Unhealthy` state for routing — the + same distinction the NATS path needs a KV "installed" marker to express. + Under K8s, health comes free (readiness gates the EndpointSlice), but the + *presence* of an optional service still has to be modeled as a resolution + outcome, not a failure. + +This is a new implementation requirement on both paths (`ServiceRegistry`'s +`Resolve`/`Watch` contract needs a way to express "absent" distinctly from +"unhealthy"), not a change to the primary/fallback recommendation. + +Two secondary observations from that lane, noted but not verdict-changing: +- **Two discovery systems, not one.** Under K8s, sync gRPC is served by + cluster DNS/EndpointSlice, while async NATS status/telemetry is served by + NATS's own clustering. These should not be conflated — this is exactly + why the `ServiceRegistry` seam matters: it isolates the sync path and + keeps NATS purely for async fan-out, as already planned. +- **Config relocation.** Moving install/run/upgrade from `config.toml` into + ConfigMaps/Secrets is a real change to how archie-core is deployed, + worth a deliberate PRD/settled-design before committing — a phased + concern, not a blocker for the discovery decision. + +## Bottom line + +Treat Kubernetes-native discovery (Helm chart + Operator on k3s / a real +cluster) as primary, and reuse NATS as the built-in fallback so single-host +no-K8s customers still get "install and auto-join." This is the only +ordering that satisfies the hardest requirement at the infrastructure +layer, keeps the hosted path free from one codebase, and doesn't force the +smallest customer into a full cluster. The two implementations live behind +one `ServiceRegistry` interface, so neither is a fork. The one added +implementation requirement is the optional-service "absent ≠ down" +resolution model on both paths. + +Full recovered K8s-lane transcript, for the record: session path +`.../cf1825c4-0c08-4aaa-9ea1-aac40893825f/run-0/session.jsonl` (external to +this repository; not reproduced here). diff --git a/docs/prds/service-decomposition.md b/docs/prds/service-decomposition.md new file mode 100644 index 000000000..c6a096976 --- /dev/null +++ b/docs/prds/service-decomposition.md @@ -0,0 +1,147 @@ +# Service decomposition -- decision + +**Status:** Proposed -- not yet approved for implementation +**Date:** 2026-09-05 +**Beads epic:** not yet filed; file one before any implementation work starts +**Prior art:** `docs/inspiration/2026-09-05-awx-service-decomposition-and-ansible-task-model.md`, +`docs/inspiration/module-reference-awx-decomposition-vs-archie.md`, +`docs/inspiration/service-discovery-research-2026-09-05.md`, +`/work/artefacts/archie-core-current-state-archify.html`, +`/work/artefacts/archie-core-ideal-state-archify.html` + +## Decision + +archie-core moves from one Go binary (`archied`) hosting every domain +in-process to a set of independently deployable services, each exposing +exactly one typed contract. Consumers depend on the contract; they never +reach into another service's internal types, session state, or config. +gRPC is the synchronous transport; NATS remains for asynchronous fan-out +(status, telemetry) rather than being replaced. + +This is a **distribution decision**, not a rewrite of Archie's domain +model. The six-domain split already approved in +`docs/architecture/index.md` and `docs/architecture/organisation.md` +(Identity, Agent, Messaging, Workflow, Work Intake, Plugin) is unchanged; +this PRD is about how the *processes* running that domain logic get +packaged, contracted, and discovered -- not about redrawing domain +boundaries. + +### Target services + +| Service | Contract it owns | Existing code it replaces/wraps | +|---|---|---| +| UI Service | thin client only; talks to Gateway's contract | `internal/webui` (SPA hosting stays; API surface becomes a Gateway client, not a peer with direct access) | +| Gateway Service | routing / turn / session contract | `internal/gateway` | +| Messaging Service | channel-plugin contract (Telegram, email, Discord, Teams, ...) | `internal/channels` | +| Curator Service | curator contract; **optional**, installs as a Gateway plugin | `internal/domain/curator` | +| Scheduler Service | cron/ticker contract | `internal/domain/scheduling` | +| Execution Environment | execution-dispatch contract | `internal/domain/workflow`, task dispatch paths | +| Runner | isolated worktree/container execution | `internal/container`, `internal/agentexec`, `internal/worktree` | +| State Store | state-access contract | `internal/store` | + +### Discovery + +Per `docs/inspiration/service-discovery-research-2026-09-05.md`: + +- **Primary:** Kubernetes-native discovery (Service + headless-Service/DNS + + EndpointSlice, served by CoreDNS), delivered as a Helm chart + Operator, + on k3s for single-host installs and a real cluster for multi-host or a + hosted offering. +- **Fallback:** NATS-based discovery (micro `$SRV` announce + JetStream KV + registry) for no-K8s single-host installs, with a custom gRPC resolver + reading `host:port` from the KV bucket. +- Both implementations sit behind one internal interface: + + ```go + type ServiceRegistry interface { + Resolve(service string) []Endpoint + Watch(service string) <-chan Event // join/leave + } + ``` + +- An optional service (Curator today; others later) that is not installed + must resolve to a distinct `NotInstalled` outcome, never a `Down`/ + `Unhealthy` one. Under K8s this falls out of DNS returning no record; + under the NATS fallback it requires an explicit "installed" marker in the + KV registry, not just absence-implies-down. + +### Distribution model this supports + +Both self-hosted (customer runs the Helm chart or the NATS-fallback +binaries themselves) and a managed/hosted offering (we run the same Helm +chart) are in scope -- this is the RHEL / Ansible Automation Platform +model: one architecture, sold both ways. This PRD does not decide the +business model; it decides that the architecture must not foreclose +either shape. + +## Why this shape + +`docs/inspiration/module-reference-awx-decomposition-vs-archie.md` +established that Archie's existing domain split already does what AWX's +service-decomposition post argues for -- boundaries exist. What doesn't +exist is deployability: `internal/webui` is not a passive frontend today. +Verified couplings that this decomposition must eliminate: + +- Telegram's task-action execution (pause/resume) currently routes through + `internal/webui`'s own HTTP handler in-process + (`chatTaskActorAdapter.ApplyChatTaskAction`, + `internal/app/archied/main.go:224-242`, dispatching into + `internal/webui/server.go:231`'s `handleTaskAction`). Under this PRD, + Messaging Service calls Gateway Service's contract directly -- no HTTP + detour through a UI process. +- `internal/webui/api_chat.go` calls `gateway.SessionStore`, `Router`, and + `TurnHistory` directly (lines 97, 142, 198, 256, 261) instead of through + a narrow interface. Under this PRD, UI Service only ever sees Gateway + Service's `ChatContract`. +- `bootstrap.go:1239` shares the daemon's live `config.Holder` with webui + rather than passing config. Under this PRD, each service owns its own + configuration surface; nothing shares a live in-process struct across a + service boundary because there is no longer a shared process. + +## Non-goals + +- Redesigning the domain model (Identity/Agent/Messaging/Workflow/Work + Intake/Plugin boundaries are settled elsewhere and unchanged). +- Replacing NATS. It remains the async transport; this PRD only adds gRPC + for synchronous request/response and (optionally) K8s-native discovery. +- Deciding the business model (self-hosted vs. managed vs. both). Kept + open deliberately; the architecture must serve either. +- Config relocation (`config.toml` -> ConfigMaps/Secrets) is real work + implied by a K8s-native deployment but is its own decision, not decided + here. Flagged in the discovery research as a phased concern. +- Any concrete adoption of Pi (or another external agent harness) as an + execution runner. Raised in conversation alongside this decomposition, + but it is a separate decision (agent execution as a strict data + boundary, `ARCHITECTURE.md`) and does not gate or get gated by this PRD. + +## Open questions (blocking before implementation starts) + +1. **Extraction order.** Which service gets pulled out of the monolith + first? The webui entanglement findings above suggest Gateway Service's + `ChatContract` should exist before UI Service is extracted, since UI is + the most entangled consumer today. Not yet sequenced. +2. **Contract definition mechanism.** Protobuf/gRPC service definitions + need a home (a new `proto/` or `contracts/` tree, versioning scheme, + codegen wiring into `Taskfile.yml`). Not yet designed. +3. **In-process vs. out-of-process during migration.** Whether services + can run multiplexed inside one binary behind the same `ServiceRegistry` + interface during a transition period (so extraction is incremental and + testable) or whether each service must be a separate binary from day + one. Strongly recommend the former for a solo maintainer, but not + decided here. +4. **State Store's contract shape.** Whether every service gets its own + storage (true microservice isolation) or several services share the + State Store service's contract (less isolation, less migration risk). + Not yet decided. +5. **Helm chart / Operator ownership.** Net-new work with no existing + analog in this repository (`deployments/` currently holds TOML profiles + and a `docker-compose.yml`, not Kubernetes manifests). Scoping not yet + started. + +## Completion criteria + +Not applicable yet -- this PRD records the target shape and the discovery +decision. It does not authorize implementation. Before work begins: file +a beads epic, resolve the open questions above (at minimum #1-#3), and +get explicit go-ahead given the scale of this change relative to +`AGENTS.md`'s "smallest workable change" default for this solo project.