[EPIC] Hosted service: resilient, multi-instance, multi-tenant deployment #1821

Open
opened 2026-11-08 14:02:48 +00:00 by sam · 1 comment
Owner

Archie is to be offered as a hosted service behind one public GitHub App, with a resilient load balancer and many instances. This epic holds the design discussion and the work it produces.

Open questions (maintainer):

  • Launch scale: tens of tenants (one stack per tenant fits today's single-owner services) or thousands (a shared multi-tenant fleet)?
  • Runtime: Kubernetes, or the current process model behind a cloud load balancer?

Constraints found in the code:

  • The State Store and Gateway each hold a Postgres advisory lock for their whole life; a second instance refuses to start. A load balancer can only fail over to a standby.
  • archied runs the scheduler, polling, review scans, the container pool on its local Docker and pushes in one process; two instances double-poll and each runs containers only on its own host.
  • Webhook intake must accept on any instance and be durable before acknowledging.
  • Tenants' agents would share hosts; Docker alone is not a tenant boundary.
  • NATS is a single server.

Already in place: orgs as the tenant boundary (Cedar policies, org-scoped credentials, task grants), NATS work queue, Postgres as the system of record.

Sequencing: items that also harden self-hosted installs (owner failover, scheduler split, NATS cluster) can land before 1 December; hosted-only items (tenant isolation, installation routing #1820) after, unless launch moves earlier.

Done when the open questions are decided and every item below is closed or deliberately deferred.

Archie is to be offered as a hosted service behind one public GitHub App, with a resilient load balancer and many instances. This epic holds the design discussion and the work it produces. Open questions (maintainer): - Launch scale: tens of tenants (one stack per tenant fits today's single-owner services) or thousands (a shared multi-tenant fleet)? - Runtime: Kubernetes, or the current process model behind a cloud load balancer? Constraints found in the code: - The State Store and Gateway each hold a Postgres advisory lock for their whole life; a second instance refuses to start. A load balancer can only fail over to a standby. - archied runs the scheduler, polling, review scans, the container pool on its local Docker and pushes in one process; two instances double-poll and each runs containers only on its own host. - Webhook intake must accept on any instance and be durable before acknowledging. - Tenants' agents would share hosts; Docker alone is not a tenant boundary. - NATS is a single server. Already in place: orgs as the tenant boundary (Cedar policies, org-scoped credentials, task grants), NATS work queue, Postgres as the system of record. Sequencing: items that also harden self-hosted installs (owner failover, scheduler split, NATS cluster) can land before 1 December; hosted-only items (tenant isolation, installation routing #1820) after, unless launch moves earlier. Done when the open questions are decided and every item below is closed or deliberately deferred.
Author
Owner

Items: #1822 decision (launch scale, platform), #1823 State Store and Gateway instances, #1824 archied coordination split, #1825 NATS cluster, #1826 webhook intake behind the LB, #1827 tenant isolation, #1820 GitHub App installation routing.

Items: #1822 decision (launch scale, platform), #1823 State Store and Gateway instances, #1824 archied coordination split, #1825 NATS cluster, #1826 webhook intake behind the LB, #1827 tenant isolation, #1820 GitHub App installation routing.
sam added the epic label 2026-11-08 14:03:35 +00:00
sam added the medium label 2026-11-12 15:51:29 +00:00
sam added a new dependency 2026-11-14 14:45:14 +00:00
sam added a new dependency 2026-11-14 14:45:14 +00:00
sam added a new dependency 2026-11-14 14:45:14 +00:00
sam added a new dependency 2026-11-14 14:45:14 +00:00
sam added a new dependency 2026-11-14 14:45:14 +00:00
sam added a new dependency 2026-11-14 14:45:14 +00:00
sam added a new dependency 2026-11-14 14:45:15 +00:00
sam added a new dependency 2026-11-14 14:45:15 +00:00
sam added the deferred label 2026-11-15 17:26:11 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sam/archie#1821