From 559e27dded241499ca0b7ea307dee96bcb36aa9b Mon Sep 17 00:00:00 2001 From: Sam Catlow Date: Sat, 19 Sep 2026 18:13:55 +1000 Subject: [PATCH] docs(prds): complete the image gen/edit PRD across all five open beads The contract-only doc (#497) explicitly deferred hosted provider (#498), local GPU provider (#499), /image routing (#500), and channel/web delivery (#501) to sibling beads with no design recorded. Fill in all four plus a call-site inventory: - Hosted: decide ai-sdk for generate, direct HTTP for edit (ai-sdk has no edit support) -- the decision the original doc explicitly punted on. - Local: ComfyUI as the baseline recipe (no prior art in this repo to reconcile), submit/poll shaped like minimax.Client.GenerateAndWait. - /image: Router.handleImage beside handleModel, per-request mechanism selection (session-scoped preference has no generic home in Router today, confirmed by reading its fields -- not invented here). - Delivery: mostly already built. generate_video's MultimodalResult -> MediaEvent -> channel pipeline already handles "image" generically; Telegram already calls SendPhoto. Documented per-channel evidence instead of assuming a gap that isn't there. --- docs/prds/image-capability-contract.md | 256 +++++++++++++++++++++---- 1 file changed, 222 insertions(+), 34 deletions(-) diff --git a/docs/prds/image-capability-contract.md b/docs/prds/image-capability-contract.md index f6ba7dd07..9e450a855 100644 --- a/docs/prds/image-capability-contract.md +++ b/docs/prds/image-capability-contract.md @@ -1,11 +1,16 @@ # Image capability contract and configuration -Epic: archie-core-1786748942120-1-6636e629 ("Image generation capability: -hosted API baseline plus local GPU workflows"). -Scope: archie-core-1786748942147-2-47cd5bd1 — the contract and configuration -only. Hosted provider implementation (gpt-image-2), local GPU backend, -`/image` routing, and delivery are separate, dependent beads and are not -built here. +Epic: archie-core-1786748942120-1-6636e629 / GitHub #496 ("Image generation +capability: hosted API baseline plus local GPU workflows"). + +The contract and configuration (#497) shipped first and is settled — see +"Package: `internal/domain/image`" below. This document now covers the full +feature set: hosted provider (#498), local GPU provider (#499), `/image` +routing and clarification (#500), delivery through channels and web (#501). +Documentation (#502) and the cross-cutting test matrix (#503) are execution +work against the design below, not open design questions, so they are not +given their own sections — each preceding section states its own test +obligations inline. ## Problem @@ -233,37 +238,220 @@ whole point of the contract being provider-neutral. - No provider implementation exists yet, so no live-call tests — those belong to the hosted/local provider beads. -## Non-goals (deferred to sibling beads) - -- Implementing any real `Provider` (hosted or local). -- `/image` clarification/routing flow and session-scoped mechanism - selection. -- Delivery through channel abstractions. -- Wiring `Registry` into `bootstrap.go` (nothing to wire until a real - provider exists to register). - -## Note for the hosted-provider bead (archie-core-1786748942169-3-ae7cfaea) +## Hosted provider (#498) `github.com/samcharles93/ai-sdk` (already a repo dependency) has its own provider-agnostic image facility: `image.Provider` (`Name`, `GenerateImage(ctx, GenerateImageRequest) (GenerateImageResponse, error)`), -`image.Client`/`core.GenerateImage`, and `registry.Registry.RegisterImage`/ -`.Image(name)` — the same shape chat model registration already uses. Three -backends implement it today (azure, togetherai, xai); OpenAI does not yet. +`image.Client`, and `registry.Registry.RegisterImage`/`.Image(name)` — the +same shape chat model registration already uses. Three backends implement it +today (azure, togetherai, xai); OpenAI does not yet. -The hosted-provider bead should adapt this instead of writing a raw HTTP -client, the same way `curatorLLMRunner` (internal/app/archied/main.go) -adapts `runtime.Runtime` for chat rather than the domain layer talking to -ai-sdk directly — `internal/infrastructure/image.Provider` wraps an -`ai-sdk/image.Provider` (or a new `ai-sdk/provider/openai` image -implementation, upstream, if one is added) behind this package's `Provider` -interface. +`internal/infrastructure/image.Provider` wraps an `ai-sdk/image.Provider` +behind this package's `Provider` interface, the same way `curatorLLMRunner` +(`internal/app/archied/main.go`) adapts `runtime.Runtime` for chat rather +than the domain layer talking to ai-sdk directly. -One real gap: ai-sdk's `image.GenerateImageRequest` has no input-image or -mask fields — text-to-image only. `EditRequest`/`Capability.Edit` in this -contract has nothing to adapt to yet. The hosted-provider bead needs either -an upstream ai-sdk change (adding edit support to the `image` package) or a -direct, non-ai-sdk client for the edit endpoint specifically, with -generation still going through ai-sdk. Decide and record which at that -bead's own design-doc step; this doc takes no position since editing was -out of scope here. +**Decision: generation goes through ai-sdk, edit does not.** ai-sdk's +`image.GenerateImageRequest` is text-to-image only — no input-image or mask +fields — so there is nothing to adapt `EditRequest`/`Capability.Edit` onto. +Blocking the entire hosted provider on an upstream ai-sdk change is not +this epic's call to make on someone else's project's timeline. Instead: + +- `Generate` wraps `ai-sdk/image.Client.GenerateImage` against whichever + backend `ImageHostedProvider.Class` names (azure/togetherai/xai today). +- `Edit` is a direct HTTP client hitting the OpenAI-compatible + `/images/edits` multipart endpoint, built the same way `minimax.Client` + (`internal/tools/minimax/minimax.go`) is a direct client for an API + ai-sdk doesn't cover — not a second abstraction layer, one `http.Client` + call with request/response mapping into this contract's `EditRequest`/ + `Result`. +- A provider that only wraps ai-sdk (no edit client configured) declares + `Capability{Generate: true, Edit: false}` truthfully rather than + returning `ErrUnsupported` at call time for something `Capability` + already said it couldn't do. +- If ai-sdk gains edit support upstream later, the direct client is deleted + and `Edit` moves onto it — recorded here so that's a known, not a + surprise, follow-up. + +**Provider error mapping** (every ai-sdk/HTTP error translated at the +adapter boundary, never leaked as-is): + +| ai-sdk / HTTP condition | contract error | +|---|---| +| `image.ErrNoProvider`, `image.ErrAuthFailed`, missing/invalid credential, backend unreachable | `ErrUnavailable` | +| `image.ErrEditNotSupported`, request needs a `Capability` the provider didn't declare | `ErrUnsupported` | +| input image over configured byte limit (checked before the call, per "Timeouts, size limits, cancellation" above) | `ErrInputTooLarge` | +| `ctx.Err() == context.DeadlineExceeded` | `ErrTimeout` | +| `ctx.Err() == context.Canceled` | `ErrCanceled` | +| `image.ErrRateLimited`, `image.ErrContentFiltered`, any other 4xx/5xx | wrapped verbatim under `ErrUnavailable` with the provider's own message as detail — a rate limit or content filter is "unavailable for this request", not a new sentinel the caller needs to branch on differently | + +**Testing**: `httptest.Server` fixtures for the edit client (request +multipart shape, response decoding, 4xx/5xx mapping); a fake `ai-sdk/image.Provider` +for the generate path so the adapter's translation is tested without a live +backend, matching #498's "no live API key needed for tests" criterion. + +## Local GPU provider (#499) + +**Decision: ComfyUI is the baseline recipe**, not a name this doc picked +arbitrarily — it is the de facto scriptable HTTP API for Stable-Diffusion-class +local generation, already shaped like the async submit/poll pattern +`minimax.Client.GenerateAndWait` established in this codebase (submit a +workflow, poll `/history/{id}` until the output node has files, download). +No other local backend is referenced anywhere in this repo or its docs, so +this decision has no competing prior art to reconcile — it is a fresh +choice, recorded here so the implementing bead does not re-litigate it. + +```go +// internal/infrastructure/image/comfyui +type Config struct { + Enabled bool + BaseURL string // e.g. http://127.0.0.1:8188 + Workflow string // path to a saved ComfyUI workflow JSON template + // (prompt/size/seed substituted at call time) + Timeout time.Duration + PollInterval time.Duration + MaxWait time.Duration +} +``` + +- `Generate`/`Edit` load `Workflow`, substitute the templated fields (prompt, + size, seed, and for `Edit` the input image as a base64 node input — ComfyUI + supports img2img/inpaint workflows natively, so `Capability.Edit`/`Mask` + can both be true for a correctly authored workflow, unlike the hosted + provider), POST to `/prompt`, then poll `/history/{prompt_id}` the same + shape MiniMax's poll loop already proves out in this codebase. +- `Capability.Sizes` is empty (ComfyUI workflows declare their own + resolution in the template, not a fixed enum) — the contract already + allows an empty `Sizes` to mean "provider does not constrain size". +- Backend-down (`BaseURL` unreachable) and workflow-missing are both + `ErrUnavailable`, not a startup failure — same "degrade the capability, + never break text chat" rule the contract already states. +- Per #499's own acceptance criteria, at least one recipe must be exercised + end-to-end on real hardware or a reproducible harness, and no model + weights are ever committed or fetched by Archie itself — the operator + points `Workflow`/`BaseURL` at their own running ComfyUI instance and + already-downloaded checkpoint, exactly like `[providers.ollama]` points at + an already-running Ollama rather than Archie managing the model. + +**Testing**: a fake HTTP server standing in for ComfyUI's `/prompt` and +`/history` endpoints (submit, poll, malformed history, timeout, job +not-found); a real-hardware smoke test documented as a manual `task` +target per #503, not part of `task check` — the same reasoning `task vuln` +and `task deadcode` are excluded from the gate. + +## `/image` routing and clarification flow (#500) + +**Call site: `internal/gateway/gateway.go`'s `Router`**, following the +`/model` shape exactly — `/model`, `/spawn`, `/approve` etc. are already +"commands handled without the LLM" (`Router.handleModel`, dispatched via +`isLocalCommand`/`parseCmd` against `messaging.LocalCommandSpecs()`). +`/image` is a new entry in `internal/domain/messaging/commands.go`'s +`localCommandSpecs`, and a new `Router.handleImage(ctx, arg)` beside +`handleModel`, gated on a new `Router.Images *image.Registry` field (nil = +"/image not configured", the same pattern `Models`/`Tasks`/`Controller` +already use — not a new gating convention). + +**Decision: mechanism selection is per-request, not session-persisted.** +The epic allows "a remembered preference only where existing session/config +conventions support it" — and checking `Router`'s actual fields +(`ModelPersist`, `Identity`, the rest) shows there is no generic per-session +key/value store the neutral command path can lean on today; `/model`'s own +`--session`/`--global` scoping lives entirely in +`internal/channels/telegram/model.go`, a Telegram-specific enrichment, not +something `Router` provides generically. Inventing a new generic +session-preference store just for `/image` would be exactly the kind of +scaffolding CLAUDE.md's Architectural Simplicity rule warns against for one +caller. So: every `/image` invocation with no explicit mechanism produces +the clarification prompt; an explicit `--hosted`/`--local` flag (mirroring +`parseModelRest`'s `--provider` flag shape) skips it for that one call. +Revisit if a second command wants session-scoped preference too — that's +the point at which shared infra earns its cost. + +**Flow**, matching `handleModel`'s reply-string shape (no separate state +machine object, since the whole exchange is "ask once, use the answer +inline in the next message" — the model-selector's own worked example): + +1. `/image ` with no `--hosted`/`--local` → reply lists the + registered classes (from `Registry.ByClass`) and asks the user to repeat + the command with `--hosted` or `--local`. No provider is called. +2. `/image --hosted ` or `--local` → resolves to + `Registry.ByClass(class)`, picks `ImageConfig.Default` if it names a + provider of that class, else the sole registered provider of that class, + else (more than one, no default) asks which named provider. +3. Edit requests (`/image --edit ` with an attached image on the + inbound message) validate against `Capability.Edit`/`MaxInputs`/`Mask` + before calling — an edit request to a generate-only provider is a + clarification reply ("this provider doesn't support editing"), not a + provider call that returns `ErrUnsupported`. +4. A successful call returns a `Result`; delivery is the next section's + concern, not this one's — `handleImage` returns the same + `(string, error)` shape `handleModel` does for the text portion, and a + separate mechanism (below) carries the actual image bytes. + +**Testing**: channel-neutral state-machine tests for `handleImage` against +a fake `*image.Registry` (mirrors `#500`'s own acceptance criterion); Telegram +adapter tests only for wiring, not for re-testing the state machine. + +## Delivery through channels and web (#501) + +This is the smallest section because most of the delivery path **already +exists and is already proven** — by `generate_video` +(`internal/tools/minimax`), which ships today. Do not rebuild it. + +The existing pipeline: a result carrying media returns +`tools.MultimodalResult{IsMultimodal: true, URLs: []tools.MediaRef{{Type: "image", ...}}}` +→ `multimodalMediaRefs` (`internal/app/archied/telegram_setup.go`) decodes +it → `TurnStream.Media(gateway.MediaEvent)` → each channel's own +`Media`/`MediaAttachment` handling. `MediaRef.Type` already documents +`"image"` as a valid value alongside `"video"`; nothing in the pipeline is +video-specific. + +**Per-channel state, checked directly rather than assumed:** + +| channel | image delivery | evidence | +|---|---|---| +| Telegram | **already works** | `internal/channels/telegram/media.go`'s `telegramMediaSender.dispatch` already branches on `att.Type` and calls `bot.SendPhoto` for `"image"` (`SendVideo` for `"video"` is the same function) | +| Dashboard (webui) | **tracked gap, not new** | `internal/webui/api_chat.go:355`'s `chatStreamSink.Media` has no inline rendering path yet and degrades to a link/error string — already tracked as archie-core-1786748942243-6-f109697e. This epic does not need to fix it, but #501's acceptance criterion ("unsupported paths report a clear limitation") is already met by the existing degrade-to-text behavior; closing the dashboard gap is that bead's job, not this one's | +| Email | **no attachment sender exists** | `grep MediaAttachment internal/channels/email` returns nothing. Out of scope for this epic per #496's non-goals (no channel-abstraction rework) — email delivery of a generated image degrades the same way the dashboard does today, which is acceptable under #501's "or unsupported paths report a clear limitation" clause | +| Webhook | not applicable | webhook is an inbound intake channel (`internal/channels/webhook`), not an outbound chat reply target — no `/image` command can originate there in the first place | + +**The only new work `internal/tools/minimax` didn't already need**: the +autonomous-tool call site. `/image` (previous section) calls a provider +directly and does not go through the model's tool-calling loop at all. +Separately, exposing `generate_image`/`edit_image` as an ordinary tool +(so an agent task can generate a diagram or illustration mid-task, not only +a human typing `/image`) is a thin wrapper — `internal/tools/image/tool.go`, +same shape as `internal/tools/minimax/tool.go`, calling +`image.Registry.Get(name).Generate/Edit` and returning the identical +`MultimodalResult` shape. Wired in `bootstrap.go` next to +`registerMinimaxTool`, gated the same way (registered only when a provider +resolved a working credential/backend, never advertised broken). This +reuses the entire delivery pipeline above with zero new plumbing — the tool +call site is what's new, not the delivery. + +**Testing**: `multimodalMediaRefs` already has coverage for the shape; +add a table case for `Type: "image"` if one doesn't already exist. The tool +wrapper gets the same fake-provider test shape as the routing flow above. +No new delivery-path tests are needed — that pipeline's tests belong to +whichever change originally proved it out. + +## Call site inventory + +Every file a full implementation touches, so no bead re-derives this by +grep: + +| concern | file | change | +|---|---|---| +| domain contract | `internal/domain/image/{contract,registry}.go` | done (#497) | +| config schema + validation | `internal/infrastructure/configuration/{types,validate}.go` | done (#497); extended here only if ComfyUI's `Workflow` field needs a new validation rule (path exists, non-empty when `Local.Enabled`) | +| hosted provider | `internal/infrastructure/image/openai/*.go` (or per `Class`) | new (#498) | +| local provider | `internal/infrastructure/image/comfyui/*.go` | new (#499) | +| `/image` command spec | `internal/domain/messaging/commands.go` | new entry in `localCommandSpecs` | +| `/image` routing | `internal/gateway/gateway.go` (`Router.Images`, `handleImage`) | new (#500) | +| autonomous tool | `internal/tools/image/tool.go` | new (#501) | +| tool registration | `internal/app/archied/bootstrap.go` (`registerImageTool`, beside `registerMinimaxTool`) | new (#501) | +| `Registry` wiring | `internal/app/archied/bootstrap.go` (construct `image.Registry`, register configured providers, hand to `Router.Images`) | new — this is the "nothing to wire until a real provider exists" step the original contract doc deferred | +| Telegram delivery | `internal/channels/telegram/media.go` | none — already handles `"image"` | +| Dashboard delivery | `internal/webui/api_chat.go` | none in this epic — tracked separately (archie-core-1786748942243-6-f109697e) | +| docs | `docs/` user-facing guide (#502) | new, against this design |