Dashboard mutations 403 behind a TLS-terminating reverse proxy: Origin check ignores X-Forwarded-Proto #1049

Closed
opened 2026-10-27 13:38:48 +00:00 by sam · 0 comments
Owner

Every operator action from the dashboard -- Stop, Cancel, Retry, Approve, Reject, Abandon, Archive -- fails with HTTP 403 in the production deployment. Observed 2026-08-24: POST https://archie.catlow.cloud/api/tasks/3/action returned 403 in 7ms with a 30-byte body. This left the operator with no way to stop a running task from the web UI at all.

Root cause

authorizeTaskMutation (internal/webui/api_tasks.go:197-221) derives the expected Origin scheme from the daemon's OWN connection:

wantScheme := "http"
if r.TLS != nil {
    wantScheme = "https"
}
...
if ... || !strings.EqualFold(u.Scheme, wantScheme) || !strings.EqualFold(u.Host, r.Host) {
    http.Error(w, "cross-origin mutation refused", http.StatusForbidden)

archied runs behind a reverse proxy that terminates TLS (confirmed: archie.catlow.cloud redirects to auth.catlow.cloud for forward auth, and the request was HTTP/2 to the proxy). The daemon therefore receives a PLAIN HTTP connection, so r.TLS == nil and wantScheme == "http", while the browser correctly sends Origin: https://archie.catlow.cloud. Scheme mismatch -> 403 on every mutation.

X-Forwarded-Proto is never consulted anywhere in the codebase (grep -rn 'X-Forwarded' internal/webui/ internal/config/ returns nothing).

Evidence the body confirms which branch fired

There are exactly two 403 paths in this handler. Their response bodies differ in length:

  • "missing CSRF header\n" = 20 bytes
  • "cross-origin mutation refused\n" = 30 bytes

The observed response was 30 B, so the Origin comparison is what rejected it, NOT the CSRF header. The dashboard is sending X-Archie-CSRF correctly.

Why this is P1

It is a total loss of the operator control surface in the supported deployment shape. It also removes the documented remedy for other faults: archie-core-0dic leaves a stopped task's row reading running, and Parked's retry action is the stated recovery for a review that fails closed (docs/prds/adversarial-self-review.md section 4) -- both assume dashboard actions work.

Note the deployment model is explicitly plural (CLAUDE.md "Deployment model": several supported shapes, systemd is not required), so "runs behind a proxy" is a first-class configuration, not an edge case.

Steps to Reproduce

  1. Serve archied behind any TLS-terminating reverse proxy (nginx, Caddy, Traefik, or a forward-auth proxy such as Authelia), with the daemon listening on plain HTTP.
  2. Load the dashboard over https and press Stop on a running task.
  3. Observe POST /api/tasks/<id>/action -> 403, body "cross-origin mutation refused".
  4. The same build works when reached directly over http, or over https terminated by the daemon itself, which is why this does not reproduce in local development.

Acceptance Criteria

Dashboard mutations succeed when archied is behind a TLS-terminating reverse proxy; the effective scheme is derived from X-Forwarded-Proto (and the effective host from X-Forwarded-Host where applicable) ONLY when the request arrives from a trusted proxy, configured explicitly -- an unconditional trust of these headers would let any client forge the Origin check and defeat the CSRF protection this function exists to provide; direct (non-proxied) deployments keep their current strict behaviour; tests cover proxied-https, direct-http, direct-https, and a spoofed X-Forwarded-Proto from an untrusted source; the config surface and its default are documented in config.example.toml and the deployment profiles.

Notes

CONFIRMED LIVE against production 2026-08-25 via Playwright, signed in as the operator.

Reproduced the dashboard's own Stop request from the loaded page:

fetch('/api/tasks/3/action', {method:'POST',
headers:{'X-Archie-CSRF':'1','Content-Type':'application/json'},
body:'{"action":"stop"}'})

Result: status 403, body exactly "cross-origin mutation refused\n" (30 bytes), with location.origin = https://archie.catlow.cloud and location.protocol = https:.

This settles which of the two 403 branches fires. The request carried a correct X-Archie-CSRF header and same-origin credentials, so it passed the CSRF check and was rejected by the Origin comparison in authorizeTaskMutation. Combined with the deployment shape (archie.catlow.cloud 302s to auth.catlow.cloud for forward auth, so TLS terminates at the proxy and the daemon sees plain HTTP), the scheme comparison is the failing term: browser Origin scheme 'https' vs the daemon's computed wantScheme 'http' from r.TLS == nil.

The dashboard confirms the operator impact at the same time: the header reads '1 running' and task #3 is still listed as running (that stale row is archie-core-0dic, a separate defect). So in production the operator currently cannot stop task #3 from either surface -- the web UI is 403'd by this bug, and chat /stop leaves the row reading running via 0dic.

Related UX gap worth its own bead rather than folding in here: the only user-visible signal was a console error, 'Failed to load resource: the server responded with a status of 403'. The dashboard surfaced nothing actionable in the UI itself, which is why this went unnoticed until devtools were opened.
DEPLOYMENT CONFIRMED from /work/infra/automation (Ansible):

roles/traefik/templates/archie.yml.j2 defines router 'archie-webui' on the 'websecure' entrypoint with 'tls: certresolver', i.e. Traefik terminates TLS. roles/traefik/defaults/main.yml:17-18 give:
traefik_archie_webui_domain: archie.catlow.cloud
traefik_archie_webui_backend_url: http://10.36.25.188:8484 <-- PLAIN HTTP

So archied receives a plain-HTTP request and computes wantScheme='http', while the browser sends Origin: https://archie.catlow.cloud. That is the whole bug.

Two details that make the fix precise and low-risk:

  1. Only the SCHEME term fails. Traefik forwards the original Host by default (passHostHeader), so u.Host == r.Host already matches. The Origin check does not need host rewriting for this deployment -- honouring the forwarded scheme is sufficient. Keep X-Forwarded-Host support optional/secondary rather than assuming it is needed.

  2. The header archied needs is ALREADY ARRIVING. Traefik sets X-Forwarded-Proto: https on proxied requests by default. Nothing in the infrastructure has to change; archied simply never reads it. That also means the fix is verifiable against this exact deployment without an infra edit.

Auth context, for whoever implements: the same router injects 'Authorization: Bearer ' via the archie-webui-token middleware, behind authelia@docker forward auth. So the trusted-proxy boundary is well defined here -- requests reaching :8484 have already passed Traefik. Whatever trust configuration is added must still DEFAULT to not trusting X-Forwarded-* , because an operator running archied directly exposed would otherwise let any client forge Origin and defeat the CSRF protection entirely.

Close reason

Shipped in 16f9456: explicitly configured trusted forwarded headers drive effective Origin validation, with direct/proxied/spoofed regression coverage; included in archied v1.18.0.


Imported from beads archie-core-440c · GitHub #627 · type bug · priority P1 · created 2026-08-24 · closed 2026-08-25

Every operator action from the dashboard -- Stop, Cancel, Retry, Approve, Reject, Abandon, Archive -- fails with HTTP 403 in the production deployment. Observed 2026-08-24: `POST https://archie.catlow.cloud/api/tasks/3/action` returned 403 in 7ms with a 30-byte body. This left the operator with no way to stop a running task from the web UI at all. ## Root cause `authorizeTaskMutation` (internal/webui/api_tasks.go:197-221) derives the expected Origin scheme from the daemon's OWN connection: wantScheme := "http" if r.TLS != nil { wantScheme = "https" } ... if ... || !strings.EqualFold(u.Scheme, wantScheme) || !strings.EqualFold(u.Host, r.Host) { http.Error(w, "cross-origin mutation refused", http.StatusForbidden) archied runs behind a reverse proxy that terminates TLS (confirmed: archie.catlow.cloud redirects to auth.catlow.cloud for forward auth, and the request was HTTP/2 to the proxy). The daemon therefore receives a PLAIN HTTP connection, so `r.TLS == nil` and `wantScheme == "http"`, while the browser correctly sends `Origin: https://archie.catlow.cloud`. Scheme mismatch -> 403 on every mutation. `X-Forwarded-Proto` is never consulted anywhere in the codebase (`grep -rn 'X-Forwarded' internal/webui/ internal/config/` returns nothing). ## Evidence the body confirms which branch fired There are exactly two 403 paths in this handler. Their response bodies differ in length: - "missing CSRF header\n" = 20 bytes - "cross-origin mutation refused\n" = 30 bytes The observed response was 30 B, so the Origin comparison is what rejected it, NOT the CSRF header. The dashboard is sending X-Archie-CSRF correctly. ## Why this is P1 It is a total loss of the operator control surface in the supported deployment shape. It also removes the documented remedy for other faults: archie-core-0dic leaves a stopped task's row reading `running`, and `Parked`'s retry action is the stated recovery for a review that fails closed (docs/prds/adversarial-self-review.md section 4) -- both assume dashboard actions work. Note the deployment model is explicitly plural (CLAUDE.md "Deployment model": several supported shapes, systemd is not required), so "runs behind a proxy" is a first-class configuration, not an edge case. ## Steps to Reproduce 1. Serve archied behind any TLS-terminating reverse proxy (nginx, Caddy, Traefik, or a forward-auth proxy such as Authelia), with the daemon listening on plain HTTP. 2. Load the dashboard over https and press Stop on a running task. 3. Observe `POST /api/tasks/<id>/action` -> 403, body "cross-origin mutation refused". 4. The same build works when reached directly over http, or over https terminated by the daemon itself, which is why this does not reproduce in local development. ## Acceptance Criteria Dashboard mutations succeed when archied is behind a TLS-terminating reverse proxy; the effective scheme is derived from X-Forwarded-Proto (and the effective host from X-Forwarded-Host where applicable) ONLY when the request arrives from a trusted proxy, configured explicitly -- an unconditional trust of these headers would let any client forge the Origin check and defeat the CSRF protection this function exists to provide; direct (non-proxied) deployments keep their current strict behaviour; tests cover proxied-https, direct-http, direct-https, and a spoofed X-Forwarded-Proto from an untrusted source; the config surface and its default are documented in config.example.toml and the deployment profiles. ### Notes CONFIRMED LIVE against production 2026-08-25 via Playwright, signed in as the operator. Reproduced the dashboard's own Stop request from the loaded page: fetch('/api/tasks/3/action', {method:'POST', headers:{'X-Archie-CSRF':'1','Content-Type':'application/json'}, body:'{"action":"stop"}'}) Result: status 403, body exactly "cross-origin mutation refused\n" (30 bytes), with location.origin = https://archie.catlow.cloud and location.protocol = https:. This settles which of the two 403 branches fires. The request carried a correct X-Archie-CSRF header and same-origin credentials, so it passed the CSRF check and was rejected by the Origin comparison in authorizeTaskMutation. Combined with the deployment shape (archie.catlow.cloud 302s to auth.catlow.cloud for forward auth, so TLS terminates at the proxy and the daemon sees plain HTTP), the scheme comparison is the failing term: browser Origin scheme 'https' vs the daemon's computed wantScheme 'http' from r.TLS == nil. The dashboard confirms the operator impact at the same time: the header reads '1 running' and task #3 is still listed as running (that stale row is archie-core-0dic, a separate defect). So in production the operator currently cannot stop task #3 from either surface -- the web UI is 403'd by this bug, and chat /stop leaves the row reading running via 0dic. Related UX gap worth its own bead rather than folding in here: the only user-visible signal was a console error, 'Failed to load resource: the server responded with a status of 403'. The dashboard surfaced nothing actionable in the UI itself, which is why this went unnoticed until devtools were opened. DEPLOYMENT CONFIRMED from /work/infra/automation (Ansible): roles/traefik/templates/archie.yml.j2 defines router 'archie-webui' on the 'websecure' entrypoint with 'tls: certresolver', i.e. Traefik terminates TLS. roles/traefik/defaults/main.yml:17-18 give: traefik_archie_webui_domain: archie.catlow.cloud traefik_archie_webui_backend_url: http://10.36.25.188:8484 <-- PLAIN HTTP So archied receives a plain-HTTP request and computes wantScheme='http', while the browser sends Origin: https://archie.catlow.cloud. That is the whole bug. Two details that make the fix precise and low-risk: 1. Only the SCHEME term fails. Traefik forwards the original Host by default (passHostHeader), so u.Host == r.Host already matches. The Origin check does not need host rewriting for this deployment -- honouring the forwarded scheme is sufficient. Keep X-Forwarded-Host support optional/secondary rather than assuming it is needed. 2. The header archied needs is ALREADY ARRIVING. Traefik sets X-Forwarded-Proto: https on proxied requests by default. Nothing in the infrastructure has to change; archied simply never reads it. That also means the fix is verifiable against this exact deployment without an infra edit. Auth context, for whoever implements: the same router injects 'Authorization: Bearer <token>' via the archie-webui-token middleware, behind authelia@docker forward auth. So the trusted-proxy boundary is well defined here -- requests reaching :8484 have already passed Traefik. Whatever trust configuration is added must still DEFAULT to not trusting X-Forwarded-* , because an operator running archied directly exposed would otherwise let any client forge Origin and defeat the CSRF protection entirely. ### Close reason Shipped in 16f9456: explicitly configured trusted forwarded headers drive effective Origin validation, with direct/proxied/spoofed regression coverage; included in archied v1.18.0. --- _Imported from beads `archie-core-440c` · GitHub #627 · type bug · priority P1 · created 2026-08-24 · closed 2026-08-25_
sam added the highbug labels 2026-10-27 13:38:48 +00:00
sam self-assigned this 2026-10-27 13:38:49 +00:00
sam closed this issue 2026-10-27 13:38:49 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1049