[bug] a dropped workflow_call_started leaves a waiting caller holding its repo baton #1675

Closed
opened 2026-10-27 13:41:20 +00:00 by sam · 0 comments
Owner

The capacity release for a wait:true workflow.call is driven by the workflow_call_started event on the daemon bus (internal/daemon/call_waits.go). events.Bus.Publish drops an event for a subscriber whose buffer is full, so under a burst of agent events the started event can be lost: the caller never suspends, keeps its global slot and its repo baton, and the callee (same repo) queues behind it. It self-heals rather than wedging permanently, because the caller hits its task wall-clock limit and is cancelled, releasing the baton, and the callee then runs; but the call fails instead of completing.

Mitigated, not closed: the watcher subscribes with a 1024 buffer and its handler is O(1) and non-blocking. A dropped finished event is harmless (the run stays counted until its process returns).

Smallest real fix: a store-backed backstop on the slow cycle rather than trusting the bus alone — a pass that asks the store which running tasks have an open call step (kind = call, running, called_execution_id != 0) and aligns the dispatcher suspension to that set. Done when a test can drop a started event and the callee still gets capacity.

Close reason

Fixed in ffacc107: watchCallWaits reconciles call waits from durable steps every second; TestCallWaitsRecoverDroppedStart drops the started event and the callee still gets capacity.


Imported from beads archie-core-yfvj · GitHub #1560 · type bug · priority P2 · created 2026-10-08 · closed 2026-10-17

The capacity release for a wait:true workflow.call is driven by the workflow_call_started event on the daemon bus (internal/daemon/call_waits.go). events.Bus.Publish drops an event for a subscriber whose buffer is full, so under a burst of agent events the started event can be lost: the caller never suspends, keeps its global slot and its repo baton, and the callee (same repo) queues behind it. It self-heals rather than wedging permanently, because the caller hits its task wall-clock limit and is cancelled, releasing the baton, and the callee then runs; but the call fails instead of completing. Mitigated, not closed: the watcher subscribes with a 1024 buffer and its handler is O(1) and non-blocking. A dropped finished event is harmless (the run stays counted until its process returns). Smallest real fix: a store-backed backstop on the slow cycle rather than trusting the bus alone — a pass that asks the store which running tasks have an open call step (kind = call, running, called_execution_id != 0) and aligns the dispatcher suspension to that set. Done when a test can drop a started event and the callee still gets capacity. ### Close reason Fixed in ffacc107: watchCallWaits reconciles call waits from durable steps every second; TestCallWaitsRecoverDroppedStart drops the started event and the callee still gets capacity. --- _Imported from beads `archie-core-yfvj` · GitHub #1560 · type bug · priority P2 · created 2026-10-08 · closed 2026-10-17_
sam added the bugmedium labels 2026-10-27 13:41:20 +00:00
sam self-assigned this 2026-10-27 13:41:20 +00:00
sam closed this issue 2026-10-27 13:41:21 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1675