Personal token create can stall until the dashboard times out #1717

Open
opened 2026-10-31 15:24:29 +00:00 by sam · 3 comments
Owner

Creating a personal API token from the dashboard twice failed with "The operation timed out". The State Store logged identity store failed: context canceled for each attempt (about 19s after the request), so the bind transaction (GetVisibleIdentity, UpsertIdentitySubject, identity event insert) was waiting until archie-ui cancelled it. The first failure came 19s after a fresh sign-in. A retry 80 minutes later succeeded at once. Afterwards Postgres showed no lock waits, no open transactions and 19 of 100 connections. StatusCounts was also cancelled at about :00:37 and :00:46 past two consecutive hours, which points at a periodic stall in the State Store or its pool rather than at the token path.

The token is shown only once, so a create that commits after the dashboard gave up leaves a token nobody holds.

Done when: the cause of the stall is found and fixed, and a create that outlives the dashboard's timeout can't leave an orphaned token.

Creating a personal API token from the dashboard twice failed with "The operation timed out". The State Store logged `identity store failed: context canceled` for each attempt (about 19s after the request), so the bind transaction (GetVisibleIdentity, UpsertIdentitySubject, identity event insert) was waiting until archie-ui cancelled it. The first failure came 19s after a fresh sign-in. A retry 80 minutes later succeeded at once. Afterwards Postgres showed no lock waits, no open transactions and 19 of 100 connections. `StatusCounts` was also cancelled at about :00:37 and :00:46 past two consecutive hours, which points at a periodic stall in the State Store or its pool rather than at the token path. The token is shown only once, so a create that commits after the dashboard gave up leaves a token nobody holds. Done when: the cause of the stall is found and fixed, and a create that outlives the dashboard's timeout can't leave an orphaned token.
Author
Owner

A live capture during a successful retry polled pg_stat_activity every 5s. Every sample caught UpsertApplyStatus and UpsertPresence commits waiting on WALSync/WALWrite, one with five of them queued on WALWrite and one waiting on a transactionid lock on apply_status. The heartbeat upserts commit constantly, so a slow fsync stalls every commit behind them. That fits a write that waits past the dashboard's 15s on a bad flush and succeeds instantly otherwise. Next: check fsync latency on the database volume, and whether apply_status and presence need a durable commit per heartbeat (synchronous_commit off for those writes, or write only on change).

A live capture during a successful retry polled pg_stat_activity every 5s. Every sample caught UpsertApplyStatus and UpsertPresence commits waiting on WALSync/WALWrite, one with five of them queued on WALWrite and one waiting on a transactionid lock on apply_status. The heartbeat upserts commit constantly, so a slow fsync stalls every commit behind them. That fits a write that waits past the dashboard's 15s on a bad flush and succeeds instantly otherwise. Next: check fsync latency on the database volume, and whether apply_status and presence need a durable commit per heartbeat (synchronous_commit off for those writes, or write only on change).
Author
Owner

c5382a847 bounds the bind to 10s and revokes a token whose bind failed, so no orphaned tokens. The stall cause is unconfirmed, most likely State Store pool wait (pgx default pool max(4, NumCPU), one connection pinned for the ownership claim, hourly sweeps). Needs pool wait metrics or pg_stat_activity on a live host.

c5382a847 bounds the bind to 10s and revokes a token whose bind failed, so no orphaned tokens. The stall cause is unconfirmed, most likely State Store pool wait (pgx default pool max(4, NumCPU), one connection pinned for the ownership claim, hourly sweeps). Needs pool wait metrics or pg_stat_activity on a live host.
Author
Owner

Live check at idle: the State Store holds 4 connections (1 pinned for the ownership lock, 3 idle), no idle-in-transaction sessions, no lock waits, well under max_connections. Pool wait isn't visible at idle; it needs a capture during an hourly sweep or at the moment a create stalls.

Live check at idle: the State Store holds 4 connections (1 pinned for the ownership lock, 3 idle), no idle-in-transaction sessions, no lock waits, well under max_connections. Pool wait isn't visible at idle; it needs a capture during an hourly sweep or at the moment a create stalls.
sam added the bugmedium labels 2026-11-14 14:45:44 +00:00
sam added the deferred label 2026-11-15 17:26:14 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1717