fix(nell): counters squat on the SDK-reserved meta: prefix, blocking all writes after restart #903

Closed
opened 2026-10-27 13:38:13 +00:00 by sam · 0 comments
Owner

TOTAL OUTAGE. No task can be queued and no event recorded; survives restarts.

internal/nell/adapter.go:946,952 store auto-increment counters at meta:task_id_counter and meta:event_id_counter. The NellDB SDK reserves the meta: prefix for its own bookkeeping (sdk/replicate.go:841 isInternalID = id[:5]=="meta:").

DocDB.reindex() skips internal ids when rebuilding the in-memory rev cache (sdk/docdb.go:128-137), but the record stays in the log. So after a restart Get returns the counter doc WITH its _rev while d.revs has no entry, and putIn rejects exactly that: 'incomingRev != "" && !exists' -> ErrConflict (sdk/docdb.go:213-217). Permanent.

Hits nextTaskID (every EnqueueIssue) and nextEventID (every InsertEvent). processNATSTask Naks on error so NATS redelivers forever.

Reproduced: internal/nell/counter_repro_test.go - open, insert, close, reopen, insert -> ErrConflict in 7ms with no daemon/NATS/forge.

SECOND DEFECT: isInternalID exists to exclude records from replication, so the two carina instances never converge their counters and would allocate COLLIDING task IDs. Masked only because allocation currently fails outright.

FIX: move the keys out of the reserved namespace (counter:task_id / counter:event_id) AND migrate on open - the old doc is readable (only Put fails), and its next_id is load-bearing because restarting at 1 would reissue live task IDs. Also teach isMetaKey (adapter.go:803) to skip the new prefix so scans do not parse a counter as a task.

REJECTED: stripping _rev before Put is one line and works, but leaves the prefix squat, leaves counters unreplicated, and turns a loud failure into silent last-write-wins.

Full writeup: docs/incident-nelldb-counter-conflict.md


Imported from beads archie-core-1786637499562-373-11ad6000 · GitHub #394 · type bug · priority P2 · created 2026-08-08 · closed 2026-08-08

TOTAL OUTAGE. No task can be queued and no event recorded; survives restarts. internal/nell/adapter.go:946,952 store auto-increment counters at meta:task_id_counter and meta:event_id_counter. The NellDB SDK reserves the meta: prefix for its own bookkeeping (sdk/replicate.go:841 isInternalID = id[:5]=="meta:"). DocDB.reindex() skips internal ids when rebuilding the in-memory rev cache (sdk/docdb.go:128-137), but the record stays in the log. So after a restart Get returns the counter doc WITH its _rev while d.revs has no entry, and putIn rejects exactly that: 'incomingRev != "" && !exists' -> ErrConflict (sdk/docdb.go:213-217). Permanent. Hits nextTaskID (every EnqueueIssue) and nextEventID (every InsertEvent). processNATSTask Naks on error so NATS redelivers forever. Reproduced: internal/nell/counter_repro_test.go - open, insert, close, reopen, insert -> ErrConflict in 7ms with no daemon/NATS/forge. SECOND DEFECT: isInternalID exists to exclude records from replication, so the two carina instances never converge their counters and would allocate COLLIDING task IDs. Masked only because allocation currently fails outright. FIX: move the keys out of the reserved namespace (counter:task_id / counter:event_id) AND migrate on open - the old doc is readable (only Put fails), and its next_id is load-bearing because restarting at 1 would reissue live task IDs. Also teach isMetaKey (adapter.go:803) to skip the new prefix so scans do not parse a counter as a task. REJECTED: stripping _rev before Put is one line and works, but leaves the prefix squat, leaves counters unreplicated, and turns a loud failure into silent last-write-wins. Full writeup: docs/incident-nelldb-counter-conflict.md --- _Imported from beads `archie-core-1786637499562-373-11ad6000` · GitHub #394 · type bug · priority P2 · created 2026-08-08 · closed 2026-08-08_
sam added the mediumbug labels 2026-10-27 13:38:13 +00:00
sam self-assigned this 2026-10-27 13:38:13 +00:00
sam closed this issue 2026-10-27 13:38:13 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#903