A delivery the channel rejects for good is retried forever, and a long notice never reaches Telegram #2016

Closed
opened 2026-11-15 23:10:21 +00:00 by sam · 1 comment
Owner

A notice longer than Telegram allows (about 5000 characters against its 4096 limit) fails with "message is too long" on every attempt. Deliveries treat that as transient and retry it indefinitely (attempt 41 seen on staging), so the notice is never seen and the retry loop never ends.

Two parts: the telegram connector splits or trims a message to the channel limit, and the connector contract lets a connector report a permanent failure so deliveries stop retrying and record it as failed where the trail shows it.

Done when: a long investigation notice arrives on Telegram, and a delivery rejected permanently stops after one attempt with its reason visible.

A notice longer than Telegram allows (about 5000 characters against its 4096 limit) fails with "message is too long" on every attempt. Deliveries treat that as transient and retry it indefinitely (attempt 41 seen on staging), so the notice is never seen and the retry loop never ends. Two parts: the telegram connector splits or trims a message to the channel limit, and the connector contract lets a connector report a permanent failure so deliveries stop retrying and record it as failed where the trail shows it. Done when: a long investigation notice arrives on Telegram, and a delivery rejected permanently stops after one attempt with its reason visible.
sam added the bughigh labels 2026-11-15 23:10:21 +00:00
sam added a new dependency 2026-11-15 23:10:29 +00:00
Author
Owner

Fixed in archie-core b48616798 and f47e60db0 (migration 00043), and archipelago a69f713 and 6d2363c.

  • Contract: a connector's Deliver returns InvalidArgument, FailedPrecondition, PermissionDenied or NotFound when the service rejects a delivery permanently. Any other status is retried. Status codes needed no new field in DeliverResponse.
  • Deliveries: a permanent rejection marks the message failed (failed_at) after that attempt, with its reason in the attempts list, so the dashboard trail shows it. Other failures keep the backoff but stop after 150 attempts, about a day with the 10-minute cap, and the reason is prefixed "gave up after N attempts".
  • Telegram: messages over the limit were already split at paragraph and line breaks on archipelago main (48019ec). The connector now marks Bot API 400, 403 and 404 errors as permanent. 401 and 429 are still retried. When a delivery goes to several chats, it fails permanently only if every chat rejected it permanently.

Verified: the changed packages pass vet, tests and lint in both repos, and task check passed on the merged archie-core tree. The retry loop has no test, because it needs the database. Not verified on staging: a long investigation notice arriving on Telegram, and a permanently rejected delivery stopping after one attempt.

Fixed in archie-core b48616798 and f47e60db0 (migration 00043), and archipelago a69f713 and 6d2363c. - Contract: a connector's Deliver returns `InvalidArgument`, `FailedPrecondition`, `PermissionDenied` or `NotFound` when the service rejects a delivery permanently. Any other status is retried. Status codes needed no new field in `DeliverResponse`. - Deliveries: a permanent rejection marks the message failed (`failed_at`) after that attempt, with its reason in the attempts list, so the dashboard trail shows it. Other failures keep the backoff but stop after 150 attempts, about a day with the 10-minute cap, and the reason is prefixed "gave up after N attempts". - Telegram: messages over the limit were already split at paragraph and line breaks on archipelago main (48019ec). The connector now marks Bot API 400, 403 and 404 errors as permanent. 401 and 429 are still retried. When a delivery goes to several chats, it fails permanently only if every chat rejected it permanently. Verified: the changed packages pass vet, tests and lint in both repos, and `task check` passed on the merged archie-core tree. The retry loop has no test, because it needs the database. Not verified on staging: a long investigation notice arriving on Telegram, and a permanently rejected delivery stopping after one attempt.
sam closed this issue 2026-11-21 03:51:37 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sam/archie#2016