docs: capture AWX/Ansible inspiration research and service-decomposition PRD

Records the AWX pluggable-architecture and Ansible task-model inspiration
thread verbatim, the verified current-vs-ideal coupling analysis (webui's
in-process HTTP task-action detour, direct gateway internals access,
shared config.Holder), the external service-discovery research
(K8s-native primary + NATS fallback behind one ServiceRegistry seam), and
a Proposed-status PRD recording the target service shape and open
questions before any implementation work is filed.
This commit is contained in:
sam committed 2026-09-05 02:57:04 +10:00
1 parent 5942fd913e
commit d92ff825d9
4 files changed
+714

No files matched your search

@@ -0,0 +1,143 @@
# Sam's message, verbatim (2026-09-05)
Source: chat message pasting an AWX forum post (AWX modernization series,
"Refactoring AWX into a Pluggable, Service-Oriented Architecture") plus a
terminal transcript and a follow-up message including Ansible's
`ansible/playbook/task.py`. Preserved verbatim below for later reference.
Image attachment (`AAP Pluggable Architecture` diagram) is described, not
reproduced, since this file is text-only.
---
[01:09:53] sam@helix /work/clones
> git clone https://github.com/ansible/ansible.git
Cloning into 'ansible'...
remote: Enumerating objects: 644471, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 644471 (delta 5), reused 2 (delta 0), pack-reused 644450 (from 3)
Receiving objects: 100% (644471/644471), 251.86 MiB | 9.60 MiB/s, done.
Resolving deltas: 100% (427812/427812), done.
[01:12:50] sam@helix /work/clones
> git clone https://github.com/ansible/awx.git
Cloning into 'awx'...
remote: Enumerating objects: 331428, done.
remote: Counting objects: 100% (399/399), done.
remote: Compressing objects: 100% (256/256), done.
remote: Total 331428 (delta 259), reused 153 (delta 143), pack-reused 331029 (from 3)
Receiving objects: 100% (331428/331428), 346.81 MiB | 12.94 MiB/s, done.
Resolving deltas: 100% (254095/254095), done.
[01:14:59] sam@helix /work/clones
---
[Image #2]
Hi folks,
In our previous post, we discussed the upcoming changes we're making to the AWX project. Today, we'd like to dive deeper into one of the most significant changes: refactoring AWX into a pluggable, service-oriented architecture.
Current State of AWX
AWX is currently structured as a monolithic Django application. It consists of several entry points, each running as individual processes, including our API, Job Dispatcher, Event Processor, and Websocket subsystem. Alongside these, AWX relies on various ancillary processes such as Redis, Rsyslog, Receptor, Nginx, Uwsgi, and Daphne.
Challenges with the Current Architecture
Over time, it has become increasingly challenging to implement changes due to the large, self-contained nature of the application. This monolithic structure has not only slowed down our development process but also made it difficult to ensure efficient code reuse across our other applications. Consequently, this impacts both our development efficiency and the ability to quickly iterate on features and fixes. As we continue to extend capabilities across Ansible Automation Platform we cannot afford these challenges and maintain upstream projects the way they are currently structured.
Proposed Changes
To address these challenges, we are beginning to transition AWX towards a more pluggable, service-oriented architecture. By breaking down the monolithic structure into smaller, modular services, we aim to enhance both flexibility and maintainability. This shift will enable us to make changes more swiftly and efficiently in AWX while also improving our ability to reuse code across various projects leveraged by Ansible Automation Platform.
Implementation Plan
Our implementation plan involves several key steps:
Identifying Modular Services: We will start by identifying the core functionalities that can be extracted into individual plugins and/or services.
Creating a Communication Framework: Establish a robust communication framework to ensure seamless interaction between these services.
Gradual Transition: Implement the new architecture in phases to allow for continuous feedback and improvement.
Community Collaboration: Engage with our community throughout the process to gather feedback and ensure the new architecture meets the needs of our users.
Future-Looking Architecture
1316×628 38.8 KB
Expected Outcomes
By refactoring AWX into a service-oriented architecture, we expect several positive outcomes:
Improved Flexibility: Easier to implement changes and add new features.
Enhanced Maintainability: Simplified codebase with clear service boundaries.
Better Code Reuse: More efficient reuse of code across different projects.
Active Community Participation: Greater opportunity for community members to contribute and shape the project.
Call to Action
We invite feedback and collaboration with us on this significant transformation by replying to this forum post.
Join the discussions on our forum, share your thoughts, and help shape the future of awx .
Conclusion
These architectural changes are a crucial step towards a more modern and scalable AWX. We look forward to your participation and support as we embark on this journey together.
Stay updated by joining The Forum, following the
News & Announcements and
awx tags.
Thank you for your continued support.
Links in this AWX update series
Blog: Upcoming Changes to the AWX Project
Streamlining AWX Releases
Refactoring AWX into a Pluggable, Service-Oriented Architecture (this post)
Upcoming changes to AWX Operator installation methods
AWX UI and credential types transitioning to the new pluggable architecture
AWX modernization: Moving forward
AWX modernization: Ansible UI
AWX modernization: Ansible Jewel
Useful links
2024-05-30 Blog: Upcoming Changes to the AWX Project
2024-07-01 Streamlining AWX Releases
The Forum: AWX topics
The Forum: Newsletter Category
---
Taking a look at this, How could we reframe ourselves to capture some of this market?
Archie is already a beast of a project, I think that it would be an excellent idea to think about cleaning up, simplifying, and unifying many of it's layers down to cleaner interfaces and even if it's starting at the most basic layers such as splitting the UI from the backend, turning the agent into a core runner, or even having 2 separate agent definitions, an AI agent and an agent runner. or. even better, an agent runner which utilises an open source project like Pi as the AI harness instead of having to code our own one.
I have Pi running now, with an immensely fast and capable model (deepseek-v4-flash-vision-exp) which is absolutely able to perform many actions at great speed, and that could assist us to getting this chunk of work done.
the idea of everything being pluggable services - making the entire solution modular, is exactly the goal I've had from the start. and it's already heading that way, but it's missing some guidance, assistance and care. I believe that a multi-agent system could help me get there, where I talk to an agent like claude, which spawns subagents to start implementation in isolation, and another agent monitors the work, assists, and carefully merges as the migration is in progress.
[01:23:10] sam@helix /work/apps/archie-core
> tree
(directory tree of archie-core omitted here -- see repository working tree at
the time of writing for the full listing; top-level shape was: AGENTS.md,
ARCHITECTURE.md, cmd/{archie-agent,archied,archie-playbooks}, deployments/,
docs/{architecture,archive,data,guides,prds,public}, examples/, extras/,
internal/{agentexec,app,channels,config,container,daemon,domain,eventbus,
events,forge,forgerpc,gate,gateway,indexing,infrastructure,installtype,
logging,memory,natsrpc,pairing,plugin,ratelimit,releaseannounce,
releaseupdate,secret,skill,skillscript,storage,store,storerpc,taskrun,
taskstate,tools,webhookguard,webui,worktree,worktreerpc,yaegiutil}, scripts/,
tools/, ui/{dist,src,test})
[01:23:12] sam@helix /work/apps/archie-core
---
One other thing I've grabbed inspiration from the Ansible project and I'm hoping to splice in - is the task.py (and likely more items) definitions; I think that being able to use existing ansible playbooks (and eventually roles/etc) within archie, would be astoundingly benefitial for engagement.
(ansible/playbook/task.py, GPLv3, Michael DeHaan et al. -- full file pasted in
chat; omitted here as reference-only source code, not reproduced verbatim in
this repository. It defines Ansible's `Task` class: FieldAttribute-based
declarative fields such as `args`, `action`, `loop`, `register`,
`changed_when`/`failed_when`/`until`, `delay`/`poll`/`retries`,
`loop_control`; `preprocess_data` which runs `ModuleArgsParser` to resolve the
legacy free-form task shape into `(action, args, delegate_to)`; `post_validate`
which templates fields through Jinja2 (`TemplateEngine`); parent-chain
attribute inheritance (`_get_parent_attribute` walking up through Block /
TaskInclude / Play); and `_resolve_conditional` for `when:` evaluation.)
---
Attached image: "AAP Pluggable Architecture" diagram -- a decomposition tree
with "AWX Decomposition" as root, branching to: Credentials, Subscription/
Billing, Auth (children: RBAC, SSO, Policy/ABAC), Communication (child:
Inter Service, and Execution (mesh) linked back into Runtime), Telemetry,
Inventory, Runtime (child: Scheduler), Content (child: Projects).
@@ -0,0 +1,126 @@
# Module reference: AWX decomposition / Ansible task model vs. Archie
Status: exploratory reference, not a design decision. Nothing here is
authoritative -- `docs/architecture/` stays the source of truth. This exists
so a later PRD (if one gets written) doesn't have to re-derive "what do we
already have vs. what AWX/Ansible are describing." See
[[2026-09-05-awx-service-decomposition-and-ansible-task-model]] for the
source material this is built from.
Two follow-on artifacts extend this line of exploration:
- Two Archify diagrams contrasting current vs. an ideal service-oriented
shape: `/work/artefacts/archie-core-current-state-archify.html` and
`/work/artefacts/archie-core-ideal-state-archify.html`. The current-state
diagram is grounded in verified coupling (webui's task-action path routes
through its own HTTP handler in-process; direct `gateway.SessionStore` /
`Router` / `TurnHistory` access; a shared `config.Holder`) -- not the
"already split" claim this document's earlier draft mistakenly made.
- [[service-discovery-research-2026-09-05]] -- externally researched
comparison of service-discovery mechanisms for the ideal-state's "install
a service, the fleet auto-aligns" requirement. Recommends Kubernetes-native
discovery as primary (Helm chart + Operator on k3s / a real cluster) with
NATS-based discovery (already a hard dependency) as the built-in fallback
for no-K8s single-host installs, both behind one `ServiceRegistry`
interface.
## Reading the two source ideas separately
The AWX post and the Ansible `Task` paste are two different asks and should
stay two different tracks:
1. **AWX's decomposition tree** is an argument for splitting a monolith
along service boundaries (Auth, Comms, Runtime, Content, ...) with a
pluggable communication mesh between them. Archie's six-domain split
(`docs/architecture/index.md`) already **is** that argument, applied
earlier and at a different grain (domain boundaries in one Go module +
NATS transport, not separate deployable services). The gap isn't "do we
need boundaries," it's "are the boundaries clean and are they enforced."
2. **Ansible's `Task`/module system** is a request for a *content* format:
let users write YAML tasks against a stable action vocabulary and have
Archie interpret them, the way Ansible interprets `- name: ... action:
module: args`. Archie has the skeleton of this already (EDA playbook
engine, Module action position) but it's deliberately narrow today.
## AWX decomposition tree -> Archie's existing domains
| AWX node | Archie equivalent today | Location |
|---|---|---|
| Auth (RBAC / SSO / Policy-ABAC) | `internal/domain/binding` (task-to-identity/config binding) + `internal/policy` (cross-cutting, zero internal imports) + forge/channel auth (Telegram pairing, webhook HMAC) | `internal/domain/binding/`, `internal/policy/`, `internal/pairing/`, `internal/webhookguard/` |
| Communication / Inter-Service | NATS (`ARCHIE_TASKS` JetStream work queue, core-NATS RPC via `_INBOX.*`) | `internal/infrastructure/eventbus/nats/`, `internal/natsrpc/`, `internal/eventbus/` (contract, zero internal imports) |
| Execution (mesh) | Container pool running isolated agent worktrees over NATS, not a mesh of remote execution nodes (Ansible's Receptor equivalent doesn't exist -- Archie's "remote execution" is a Docker container, not a fleet) | `internal/container/pool.go`, `internal/agentexec/`, `internal/worktree/` |
| Runtime / Scheduler | `internal/domain/scheduling` (ticker contract) + cron-driven work intake | `internal/domain/scheduling/`, `internal/infrastructure/cronstore/` |
| Content / Projects | Nearest analogue is the **EDA playbook + module system** (below) plus `internal/skill` (skill catalog) -- Archie has no "project" = git-checkout-of-automation-content concept the way AWX does | `internal/domain/eda/playbook/`, `internal/domain/eda/module/`, `internal/skill/` |
| Telemetry | `internal/logging` (task sink, feed, registry) + `internal/events` | `internal/logging/`, `internal/events/` |
| Inventory | No direct equivalent. Closest is forge-side repo/identity config (`[[identities]]`, `internal/forge`) -- Archie targets one identity acting across a small set of forge repos, not a dynamic inventory of managed hosts | `internal/forge/`, `internal/config/` |
| Credentials | `internal/secret` (engine host, env/age/sops/vault engines) | `internal/secret/`, `internal/secret/enginehost/` |
| Subscription/Billing | No equivalent -- Archie is self-hosted, single-tenant; this AWX node doesn't map to anything and shouldn't | -- |
| UI split from backend | Already true: `internal/webui` (Go API server) serves a separately built SPA from `ui/dist` (Vite/React-ish, `ui/src/<feature>/`) over HTTP/SSE. There is no shared-process coupling to undo. | `internal/webui/`, `ui/src/` |
**Read on "reframe to capture market":** the honest gap next to AWX/AAP isn't
domain boundaries -- Archie already has those, arguably cleaner (env-enforced
gates, one dependency direction, contracts with zero internal imports). The
gap is (a) no content/inventory layer letting non-programmers hand Archie
declarative automation the way an Ansible playbook does, and (b) everything
still ships as one Go binary + one container image, not independently
deployable/upgradable services. (a) is the more differentiated, more
buildable opportunity given a one-person team; (b) is expensive AWX-style
service splitting with limited payoff at Archie's current scale and should
not be pursued for its own sake.
## Ansible `Task` model -> Archie's EDA playbook/module system
Ansible's `task.py` is the runtime representation of one YAML task after
`ModuleArgsParser` resolves the legacy free-form shape and `TemplateEngine`
renders Jinja2. Archie's nearest shipped analogue is the EDA playbook engine,
which is **intentionally much narrower right now**:
| Ansible concept | Archie's current shape | Gap / what porting the idea would mean |
|---|---|---|
| `Task` (one step, `action` + `args`) | `playbook.Action` (`internal/domain/eda/playbook/playbook.go`) -- but **exactly one action per playbook, position must be `"workflow"`**, hard-rejected at load otherwise | Multi-action, multi-position playbooks are the literal ask ("use existing ansible playbooks... within archie") but are explicitly blocked today on unresolved execution-time gaps: mid-run failure semantics, idempotency for non-workflow actions (see the package doc comment and `docs/architecture/plugins-and-extensions.md`'s Event sources section). This is a known, named boundary, not an oversight. |
| Module (`ansible.builtin.*`, arbitrary Python) | `module.Kind` registry, Yaegi-interpreted `.go` files against a generated per-kind symbol table (`internal/domain/eda/module/`) -- **one shipped kind: `log`** | Ansible modules are dynamically loaded from a huge ecosystem; Archie's Module position is closed-registry-of-named-kinds, each with its own hand-written contract package (`internal/domain/eda/module/<kind>`). Adding "run a shell command," "call a forge API," "send a channel message" module kinds is possible under the existing pattern but each is new work, not automatic from having the mechanism. |
| `when:` conditional | `Action.When`, compiled CEL (`internal/domain/eda/expr`), not Jinja2 | Already decided: CEL over Jinja2 (open question 1 in the playbook engine PRD, resolved). Don't reopen. |
| `register` / fact accumulation across tasks | Not present -- `DispatchInput.Event` is a flat map exposed to CEL, there's no cross-action result chaining because there's only one action | Needed before multi-action playbooks are meaningful; tracked implicitly by the same "unresolved execution-time gaps" blocker above. |
| `loop` / `until`/`retries`/`delay` | Not present | Same -- follows from allowing more than one action per playbook. |
| Field-attribute inheritance (task -> block -> play) | No block/parent chain -- a playbook is flat: one `Trigger` + one `Action` | Ansible's play/block/role hierarchy has no Archie counterpart yet; would only matter once multi-action/roles are in scope. |
| Roles (reusable task bundles) | Nearest is `internal/skill` (skill catalog, frontmatter, tool defs) -- conceptually closer to Claude Skills than Ansible roles | Not a role system; skills are agent-facing capability bundles, not declarative task sequences. |
| Channel / Forge action positions | Named and designed but explicitly **not implemented** (`docs/architecture/plugins-and-extensions.md`: "Channel and Forge action positions are designed but not yet implemented," open investigation `archie-core-t2db.19`) | This is the direct next step toward "playbooks that do things," independent of the Ansible-compatibility idea. |
**What porting real Ansible playbooks would actually require**, in rough
dependency order, if this becomes a real proposal later:
1. Lift the playbook engine's hard boundary: multi-action playbooks, which
needs the mid-run failure/idempotency semantics the doc comment flags as
the blocker today.
2. A result-passing mechanism between actions (Ansible's `register`) --
currently nothing carries one action's output into the next action's
`when`/`args`.
3. Loop/retry primitives per action.
4. A much larger Module `Kind` registry, or a deliberate decision to accept
a narrower "Archie automation language inspired by Ansible's shape" rather
than "runs unmodified Ansible YAML" -- the latter also drags in Jinja2,
the module_utils Python runtime, and inventory/host-targeting concepts
Archie has no use for (Archie targets git repos and forge issues, not
fleets of managed hosts). Recommend treating "Ansible-*inspired* syntax,
not Ansible-*compatible* execution" as the working assumption if/when a
PRD gets written -- it avoids importing Ansible's inventory/host model,
which doesn't fit Archie's single-repo-per-task execution shape at all.
## On the multi-agent / Pi-harness-as-runner idea
Out of scope for this reference doc (it's an operating-model question, not a
module-mapping one) but noted since it was raised alongside the above: this
is a `migration-decisions.md`-track question (agent execution as a strict
data boundary is a named invariant in `ARCHITECTURE.md`), not something to
back into via the EDA/module work. Flag separately if it's worth pursuing.
## Bottom line
- No code changes implied by this document. It's a map, not a proposal.
- The two ideas in Sam's message decompose cleanly: AWX's service-split
argument is mostly already satisfied by Archie's domain architecture; the
Ansible-playbook-content idea is a real, currently-blocked extension of
the EDA Module/Channel/Forge action positions, gated on the same
execution-semantics work already named in `plugins-and-extensions.md`.
- If this becomes a real initiative, it belongs in `docs/prds/` as a
decisive 1-pager per `AGENTS.md`'s scope-discipline rule -- not as an
elaboration of this file.
@@ -0,0 +1,298 @@
# Service discovery research for the archie-core service decomposition
Status: exploratory research, not a settled decision. This backs the "ideal
state" service-oriented sketch in
[[module-reference-awx-decomposition-vs-archie]] and its two Archify
diagrams (`/work/artefacts/archie-core-current-state-archify.html`,
`/work/artefacts/archie-core-ideal-state-archify.html`). Nothing here commits
the codebase to anything; if the service decomposition itself gets approved,
this is the input to the actual `docs/architecture/migration-decisions.md`
entry and/or a `docs/prds/` design doc, not a substitute for either.
Produced by an external research pass (Pi subagents plus supervisor-run
`deep_search_exa` queries against primary sources: NATS discovery, K8s-native
discovery, Consul operational footprint, gRPC/xDS/mesh weight). The K8s lane
detached mid-flight and was recovered via supervisor reply, covered from
fetched Kubernetes docs (Service/EndpointSlice/CoreDNS/Operator + k3s Helm).
Reproduced here close to verbatim as the record of that research; light
formatting only.
Business framing this research was scoped against, per Sam (2026-09-05): the
decomposition is being evaluated as a real product distribution strategy —
possibly sold self-hosted, possibly as a managed/hosted offering (the
RHEL/Ansible Automation Platform model: same product, sold both ways), not
ruled to either shape. Evaluate infrastructure choices on technical merit for
that audience, not through the lens of Sam's own homelab deployment
preferences (see `feedback_dont_project_homelab_prefs_onto_product` in
Claude's project memory). "Cost" in this document means operational
complexity, never dollar price — pricing and SaaS-plan questions are
explicitly out of scope here.
---
## 1. Candidate mechanisms
| # | Mechanism | Who stands it up | Health-aware endpoints? | "Install → auto-join" mechanics | Single-host fit | Multi-host fit |
|---|---|---|---|---|---|---|
| 1 | **Kubernetes-native** (Service + EndpointSlice + CoreDNS), delivered via Helm chart + Operator, on k3s (single-node) or a real cluster | A whole (even if small) control plane: api-server, etcd, scheduler, controller-manager, CoreDNS, kube-proxy. k3s packages this into a ~one-command install | Yes, free — readiness probes gate which pods land in an EndpointSlice | Deploy/helm install a Service (or an Operator reconciles a CR) → CoreDNS serves the name → existing services dial it | Strong (k3s is ~½ GB RAM) | Good (real cluster, or k3s server+agents) |
| 2 | **NATS-based discovery** (micro `$SRV` announce + JetStream KV registry) | NATS broker — already a hard dependency, so nothing new to stand up | No — only connection liveness; you write heartbeat+TTL for KV | Service connects to the bus on start, publishes/announces → peers observe the join | Excellent | Workable but requires a real NATS cluster / leaf nodes (TLS/NKey/JWT weight) |
| 3 | **Consul** (registry + agent + DNS) | A second control plane: 3–5 Raft servers + a client agent on every node | Yes (TCP/HTTP/gRPC/TTL) | Agent registers with server; peers resolve via Consul DNS | OK (one server + agents) | Good (multi-DC, WAN federation) but per-node agent burden |
| 4 | **etcd** (discovery substrate) | A Raft KV cluster you operate | Not natively — you own catalog/health semantics | Lease registration + prefix watch | OK (embeddable) | Good (R3 replication) but you build the semantics |
| 5 | **gRPC xDS / service mesh** (Istio/Linkerd/Traffic Director, proxyless gRPC) | A control plane (+ per-pod sidecar if mesh) | Yes (xDS health) | Register with control plane → EDS pushed to subscribed clients | Heavy / overkill | Heavy; K8s-first; sidecar cost (~0.2 vCPU + 60 MB/pod @1k rps in Istio-sidecar-land) |
| 6 | **DNS-SRV / mDNS** | Nothing new | No native health; TTL caching breaks fast failover | Register a record | mDNS yes (but link-local); DNS-SRV needs a resolver you control | mDNS fails at the first router/VLAN; DNS-SRV needs your own resolver + a custom gRPC resolver (grpc-go doesn't consume SRV natively) |
## 2. What this narrows to
Six candidates surfaced, but the field collapses to two real finalists once
three filters specific to archie-core apply: (a) the fleet is ~7 Go
services, (b) a gRPC-sync + NATS-async mix is already the working
assumption, and (c) the defining requirement is "customer installs a
service, the fleet auto-aligns, no redeploy/reconfig."
- **Consul / etcd are dominated by K8s-native.** Consul adds a whole second
control plane (Raft server quorum + a client agent on every node you don't
control) — redundant if the fleet lives in K8s (buying a second control
plane to replace one you already have), heavy if it doesn't. etcd makes
you own catalog + health semantics, which is exactly what K8s-native and
NATS each give for free in different ways. Both remain viable, just not
better.
- **gRPC xDS / service mesh is the wrong tool at this scale.** Field
consensus (multiple 2026 sources): don't adopt a mesh until 20+ services;
archie-core is at ~7, single-language (Go), with resilience logic already
in Go. xDS/proxyless-gRPC earns its keep only with locality-aware routing
/ canary / circuit-breaking across hosts, which nothing here signals. The
durable takeaway isn't "use xDS"; it's that gRPC is resolver plumbing, not
a registry — whatever registry gets picked feeds gRPC through a resolver
(DNS resolver, or a custom one).
- **DNS-SRV / mDNS are insufficient.** mDNS is link-local (dies at
router/VLAN/container boundaries), so it can't span routed multi-host;
DNS-SRV has no health and grpc-go's default resolver doesn't consume SRV
natively anyway.
So the genuine decision is K8s-native vs. reuse-NATS.
## 3. K8s-native vs. NATS: an honest weigh
**Where K8s-native wins decisively:**
- The only mechanism that delivers "install a service → the fleet
auto-aligns" at the infrastructure layer, with health-filtered endpoints,
where the customer's install is a straightforward "apply a Deployment +
Service" (or a Helm chart, or an Operator reconciling a CR). This is the
single hardest, most product-defining requirement, and K8s solves it
natively.
- Delivers, as a side effect, almost everything a 7-service fleet needs
anyway: health (readiness/liveness probes), orchestration
(restart/scale), config (ConfigMaps/Secrets), rollouts, clean external
exposure (Ingress/Gateway). Not extra weight for its own sake — buying
pieces that would otherwise be hand-built.
- The hosted/managed path is free. If the product is ever run for customers
(the "sold both ways" shape kept open above), K8s runs anyway.
K8s-native discovery makes self-hosted and hosted deployments use
identical discovery semantics — one codebase, no divergence. This is the
RHEL/Ansible Automation Platform lesson: same product, sold both ways,
same internal architecture.
**Where NATS wins:**
- Zero new infrastructure. NATS is already a hard dependency and already in
the deployment (embedded, or the existing `docker-nats-stack.toml`). For
the lowest-end single-host customer who wants "one binary dropped on a
box," NATS discovery means no cluster, no K8s — just the bus already
running.
- Genuinely idiomatic: a service connects, announces via the micro `$SRV`
tree (PING/INFO/STATS) or a JetStream KV record, and disappears the
moment its connection closes. Re-join = "open the connection and
publish." No per-service config, no sidecar, no hand registration.
**Where NATS genuinely falls short for this product:**
1. It's a NATS service locator, not a gRPC endpoint registry. The `$SRV`
framework and the KV bucket hand you names/responders, not `host:port`
gRPC endpoints. Each service's `host:port` has to be carried in
metadata/KV and a custom gRPC resolver hand-written; gRPC's built-in
resolver/health model won't "just work." Real integration work owned in
the sync path — now the product's primary contract.
2. No native health. NATS has no last-will or probe. Heartbeat+TTL (KV)
must be implemented to distinguish "installed but unhealthy" from "not
installed," and stale entries are your problem. K8s readiness probes do
this for free.
3. Discovery is coupled to broker availability. If NATS is down, every
service looks down (a hard SPOF unless NATS is itself clustered).
Multi-host is where the cost shows: a real NATS cluster or leaf nodes,
with TLS/NKey/JWT credential management — operator weight, not "just
works."
The balance: NATS is a legitimate fallback and a genuine part of the
answer; it is not a substitute for the registry that resolves gRPC
endpoints with health. If the product's sync path is gRPC, the registry
serving it should be K8s-native (DNS → resolver, headless-Service SRV for
ports, EndpointSlice health) rather than a hand-built NATS registry.
## 4. Recommendation
**Primary:** Kubernetes-native service discovery (Service +
headless-Service/DNS + EndpointSlice, served by CoreDNS), delivered as a
Helm chart + Operator, on k3s for single-host customers and a real cluster
for multi-host / hosted.
**Fallback:** NATS-based discovery (micro `$SRV` announce + JetStream KV
registry) for no-K8s single-host installs, using a custom gRPC resolver
that reads `host:port` from the KV bucket.
**Rationale, short version:**
- K8s-native is the only candidate that solves the hardest requirement —
"install a service and the rest of the system joins in to align" — at
the infrastructure layer with health-filtered endpoints and a
customer-comprehensible install. Not extra weight for its own sake; it
replaces a stack of pieces a 7-service product needs anyway, and makes
the multi-host + hosted paths free from one codebase.
- k3s packages the control plane into a ~one-command install a customer can
stand up, and the Operator/Helm/CRD pattern is exactly the "install
Curator, Gateway picks it up" product story.
- NATS is not discarded — it stays in the fleet for async fan-out, and
becomes the second discovery implementation so the smallest no-K8s
customer keeps "one binary on a box." The fallback is the honest
acknowledgment that a full K8s cluster is genuinely heavy for the very
smallest single-node customer.
Why not NATS-only: it's a service locator needing hand-built gRPC endpoint
resolution + health, and it couples discovery to broker availability. Why
not Consul/etcd: a redundant second control plane (if K8s) or "you own the
semantics" (etcd), with per-node agents. Why not a mesh: overkill at ~7 Go
services; consensus is 20+.
### The seam that keeps both shapes on one codebase
The Gateway (and any service that routes to peers) should talk to discovery
through one internal interface —
```go
type ServiceRegistry interface {
Resolve(service string) []Endpoint
Watch(service string) <-chan Event // join/leave
}
```
— with two implementations: a K8s one (DNS / EndpointSlice / custom
resolver) and a NATS one (KV / `$SRV`). The rest of the code is
registry-agnostic. This is what makes "K8s-native primary, NATS fallback" a
coherent architecture rather than a fork: business logic doesn't change,
only the registry injection point does.
## 5. End-to-end: "install Curator, Gateway discovers it"
**Path A — K8s-native (primary):**
1. Customer installs the product (`helm install archie`, or pre-provisioned
on k3s). Base services (Gateway, UI, Messaging, Scheduler, Execution,
State Store) each deploy as a Deployment + Service; Gateway is always up
and already holds a resolver for every known service name.
2. Customer installs Curator: `helm install archie-curator`, or the
Operator watches a `Curator` Custom Resource and reconciles it — creating
a Deployment (pods) + a `curator` Service.
3. The Service's label selector matches the Curator pods; the endpoint
controller writes their IPs into an EndpointSlice. Readiness probes gate
which pods are "ready" (a Curator that only registers once healthy won't
receive traffic).
4. CoreDNS immediately serves `curator.<ns>.svc.cluster.local` (A record; a
headless Service additionally gives an SRV record with the port).
5. The Gateway resolves `curator.<ns>.svc.cluster.local` through its gRPC
DNS resolver (or a K8s-aware resolver). On the next re-resolution/update
it picks up the new endpoint. No Gateway redeploy, no Gateway config
change — and because it's DNS, the Gateway code doesn't even need to know
it's running under K8s.
6. Removal: `helm uninstall archie-curator` (or scale to 0 / delete the CR)
→ EndpointSlice empties → DNS record disappears → Gateway's resolver
drops it. Clean.
**Path B — NATS fallback (single-host, no K8s):**
1. One NATS (embedded, or external via the existing compose stack) is the
shared bus; its URL is part of the base install.
2. Every service, on start, connects to that URL and either (a) registers
via the micro framework (`AddService("curator", "1.0.0", …)`), appearing
on the `$SRV` tree, or (b) writes its `host:port` + version into a
JetStream KV bucket (`archie/registry/curator/<id>`) and heartbeats it
with a TTL.
3. The Gateway holds a live roster (from `$SRV.INFO` or a KV watch) and
feeds it into a custom gRPC resolver mapping service-name → `host:port`.
4. Customer starts one Curator binary (`archie-curator`, or `docker run`
the image) with the same NATS URL in its config. It connects, announces,
and heartbeats. The Gateway's roster observes the new entry (within one
announce/heartbeat window) and starts routing to it. No Gateway restart,
no config change beyond the shared NATS URL already present in the base
install.
5. Removal: stop Curator → its connection closes → it leaves `$SRV`
immediately and its KV record expires via TTL. The Gateway routes that
capability to "not installed."
## 6. Failure-mode honesty
- **K8s-native:** pod death → readiness fails → dropped from EndpointSlice
within seconds, so the gateway stops routing to it (health-aware). DNS
TTL caching is mostly harmless because a Service's ClusterIP is stable
and all churn lives behind it in the EndpointSlice. If the control plane
goes down, already-running pods keep serving and DNS still resolves from
cache; only new scheduling/propagation halts. Known gap: cross-cluster
and off-cluster consumers (client-side resolvers, a worker fleet on VMs)
need ExternalDNS or a bridge. mTLS/auth, if needed, is a separate concern
(NetworkPolicy / cert-manager), not the mesh.
- **NATS fallback:** service dies → dropped from `$SRV` immediately, KV
record lingers until heartbeat-TTL (bounded staleness). Broker
unreachable → every service appears down (hard SPOF unless NATS is
clustered). Plain subjects aren't a consensus store (split-brain across
partitions possible); JetStream KV gives Raft per stream but KV placement
across hosts needs its own decisions. Health semantics must be built by
hand.
Both paths are robust against "a service dies"; the difference is who does
the health checking — K8s readiness probes do it for free, NATS makes you
build it.
## 7. Addendum — recovered K8s lane, optional-service semantics
A separately recovered K8s-focused research lane landed independently on
the same verdict (K8s-native as the canonical, declarative, self-healing
fleet-discovery answer; fits multi-host and the AAP-style "sold both ways"
model best) and confirmed the same install → Operator → CoreDNS → resolver
flow above. It sharpened one point specific to archie-core's shape:
**The optional, intermittently-running Curator needs explicit "absent"
semantics under K8s.** A scaled-to-zero or not-yet-created Curator has no
ready endpoint and no DNS record — the Gateway's gRPC `dns://` resolver
gets `NXDOMAIN`/empty resolution, and this must be treated as "Curator is
disabled," not "Curator is broken." Concretely:
- Lazy, on-demand re-resolution — don't pin a dead subchannel as a hard
failure.
- A distinct `NotInstalled` vs. `Down`/`Unhealthy` state for routing — the
same distinction the NATS path needs a KV "installed" marker to express.
Under K8s, health comes free (readiness gates the EndpointSlice), but the
*presence* of an optional service still has to be modeled as a resolution
outcome, not a failure.
This is a new implementation requirement on both paths (`ServiceRegistry`'s
`Resolve`/`Watch` contract needs a way to express "absent" distinctly from
"unhealthy"), not a change to the primary/fallback recommendation.
Two secondary observations from that lane, noted but not verdict-changing:
- **Two discovery systems, not one.** Under K8s, sync gRPC is served by
cluster DNS/EndpointSlice, while async NATS status/telemetry is served by
NATS's own clustering. These should not be conflated — this is exactly
why the `ServiceRegistry` seam matters: it isolates the sync path and
keeps NATS purely for async fan-out, as already planned.
- **Config relocation.** Moving install/run/upgrade from `config.toml` into
ConfigMaps/Secrets is a real change to how archie-core is deployed,
worth a deliberate PRD/settled-design before committing — a phased
concern, not a blocker for the discovery decision.
## Bottom line
Treat Kubernetes-native discovery (Helm chart + Operator on k3s / a real
cluster) as primary, and reuse NATS as the built-in fallback so single-host
no-K8s customers still get "install and auto-join." This is the only
ordering that satisfies the hardest requirement at the infrastructure
layer, keeps the hosted path free from one codebase, and doesn't force the
smallest customer into a full cluster. The two implementations live behind
one `ServiceRegistry` interface, so neither is a fork. The one added
implementation requirement is the optional-service "absent ≠ down"
resolution model on both paths.
Full recovered K8s-lane transcript, for the record: session path
`.../cf1825c4-0c08-4aaa-9ea1-aac40893825f/run-0/session.jsonl` (external to
this repository; not reproduced here).
+147
View File
@@ -0,0 +1,147 @@
# Service decomposition -- decision
**Status:** Proposed -- not yet approved for implementation
**Date:** 2026-09-05
**Beads epic:** not yet filed; file one before any implementation work starts
**Prior art:** `docs/inspiration/2026-09-05-awx-service-decomposition-and-ansible-task-model.md`,
`docs/inspiration/module-reference-awx-decomposition-vs-archie.md`,
`docs/inspiration/service-discovery-research-2026-09-05.md`,
`/work/artefacts/archie-core-current-state-archify.html`,
`/work/artefacts/archie-core-ideal-state-archify.html`
## Decision
archie-core moves from one Go binary (`archied`) hosting every domain
in-process to a set of independently deployable services, each exposing
exactly one typed contract. Consumers depend on the contract; they never
reach into another service's internal types, session state, or config.
gRPC is the synchronous transport; NATS remains for asynchronous fan-out
(status, telemetry) rather than being replaced.
This is a **distribution decision**, not a rewrite of Archie's domain
model. The six-domain split already approved in
`docs/architecture/index.md` and `docs/architecture/organisation.md`
(Identity, Agent, Messaging, Workflow, Work Intake, Plugin) is unchanged;
this PRD is about how the *processes* running that domain logic get
packaged, contracted, and discovered -- not about redrawing domain
boundaries.
### Target services
| Service | Contract it owns | Existing code it replaces/wraps |
|---|---|---|
| UI Service | thin client only; talks to Gateway's contract | `internal/webui` (SPA hosting stays; API surface becomes a Gateway client, not a peer with direct access) |
| Gateway Service | routing / turn / session contract | `internal/gateway` |
| Messaging Service | channel-plugin contract (Telegram, email, Discord, Teams, ...) | `internal/channels` |
| Curator Service | curator contract; **optional**, installs as a Gateway plugin | `internal/domain/curator` |
| Scheduler Service | cron/ticker contract | `internal/domain/scheduling` |
| Execution Environment | execution-dispatch contract | `internal/domain/workflow`, task dispatch paths |
| Runner | isolated worktree/container execution | `internal/container`, `internal/agentexec`, `internal/worktree` |
| State Store | state-access contract | `internal/store` |
### Discovery
Per `docs/inspiration/service-discovery-research-2026-09-05.md`:
- **Primary:** Kubernetes-native discovery (Service + headless-Service/DNS
+ EndpointSlice, served by CoreDNS), delivered as a Helm chart + Operator,
on k3s for single-host installs and a real cluster for multi-host or a
hosted offering.
- **Fallback:** NATS-based discovery (micro `$SRV` announce + JetStream KV
registry) for no-K8s single-host installs, with a custom gRPC resolver
reading `host:port` from the KV bucket.
- Both implementations sit behind one internal interface:
```go
type ServiceRegistry interface {
Resolve(service string) []Endpoint
Watch(service string) <-chan Event // join/leave
}
```
- An optional service (Curator today; others later) that is not installed
must resolve to a distinct `NotInstalled` outcome, never a `Down`/
`Unhealthy` one. Under K8s this falls out of DNS returning no record;
under the NATS fallback it requires an explicit "installed" marker in the
KV registry, not just absence-implies-down.
### Distribution model this supports
Both self-hosted (customer runs the Helm chart or the NATS-fallback
binaries themselves) and a managed/hosted offering (we run the same Helm
chart) are in scope -- this is the RHEL / Ansible Automation Platform
model: one architecture, sold both ways. This PRD does not decide the
business model; it decides that the architecture must not foreclose
either shape.
## Why this shape
`docs/inspiration/module-reference-awx-decomposition-vs-archie.md`
established that Archie's existing domain split already does what AWX's
service-decomposition post argues for -- boundaries exist. What doesn't
exist is deployability: `internal/webui` is not a passive frontend today.
Verified couplings that this decomposition must eliminate:
- Telegram's task-action execution (pause/resume) currently routes through
`internal/webui`'s own HTTP handler in-process
(`chatTaskActorAdapter.ApplyChatTaskAction`,
`internal/app/archied/main.go:224-242`, dispatching into
`internal/webui/server.go:231`'s `handleTaskAction`). Under this PRD,
Messaging Service calls Gateway Service's contract directly -- no HTTP
detour through a UI process.
- `internal/webui/api_chat.go` calls `gateway.SessionStore`, `Router`, and
`TurnHistory` directly (lines 97, 142, 198, 256, 261) instead of through
a narrow interface. Under this PRD, UI Service only ever sees Gateway
Service's `ChatContract`.
- `bootstrap.go:1239` shares the daemon's live `config.Holder` with webui
rather than passing config. Under this PRD, each service owns its own
configuration surface; nothing shares a live in-process struct across a
service boundary because there is no longer a shared process.
## Non-goals
- Redesigning the domain model (Identity/Agent/Messaging/Workflow/Work
Intake/Plugin boundaries are settled elsewhere and unchanged).
- Replacing NATS. It remains the async transport; this PRD only adds gRPC
for synchronous request/response and (optionally) K8s-native discovery.
- Deciding the business model (self-hosted vs. managed vs. both). Kept
open deliberately; the architecture must serve either.
- Config relocation (`config.toml` -> ConfigMaps/Secrets) is real work
implied by a K8s-native deployment but is its own decision, not decided
here. Flagged in the discovery research as a phased concern.
- Any concrete adoption of Pi (or another external agent harness) as an
execution runner. Raised in conversation alongside this decomposition,
but it is a separate decision (agent execution as a strict data
boundary, `ARCHITECTURE.md`) and does not gate or get gated by this PRD.
## Open questions (blocking before implementation starts)
1. **Extraction order.** Which service gets pulled out of the monolith
first? The webui entanglement findings above suggest Gateway Service's
`ChatContract` should exist before UI Service is extracted, since UI is
the most entangled consumer today. Not yet sequenced.
2. **Contract definition mechanism.** Protobuf/gRPC service definitions
need a home (a new `proto/` or `contracts/` tree, versioning scheme,
codegen wiring into `Taskfile.yml`). Not yet designed.
3. **In-process vs. out-of-process during migration.** Whether services
can run multiplexed inside one binary behind the same `ServiceRegistry`
interface during a transition period (so extraction is incremental and
testable) or whether each service must be a separate binary from day
one. Strongly recommend the former for a solo maintainer, but not
decided here.
4. **State Store's contract shape.** Whether every service gets its own
storage (true microservice isolation) or several services share the
State Store service's contract (less isolation, less migration risk).
Not yet decided.
5. **Helm chart / Operator ownership.** Net-new work with no existing
analog in this repository (`deployments/` currently holds TOML profiles
and a `docker-compose.yml`, not Kubernetes manifests). Scoping not yet
started.
## Completion criteria
Not applicable yet -- this PRD records the target shape and the discovery
decision. It does not authorize implementation. Before work begins: file
a beads epic, resolve the open questions above (at minimum #1-#3), and
get explicit go-ahead given the scale of this change relative to
`AGENTS.md`'s "smallest workable change" default for this solo project.