Daemon: every task/event write fails with 'sdk: revision conflict' after NellDB v0.3.0 bump — counter RMW permanently broken #602

Closed
opened 2026-10-27 13:36:59 +00:00 by sam · 0 comments
Owner

Symptom (observed 2026-08-05, task dev session)

  • Every 1m poll logs issue queued for the same open issue (sam/archie-core#8) again.
  • Every attempt logs event sink insert failed: err="sdk: revision conflict" (cmd/archied/main.go:373, the event-sink goroutine calling st.InsertEvent).
  • The daemon's own task consumer logs nats enqueue failed: err="sdk: revision conflict" (internal/daemon/daemon.go:599, calling Store.EnqueueIssue) and NAKs; JetStream redelivers to max-deliver 3 (triples of errors every ~2 min in the log).
  • The task doc is never persisted, so the enqueue dedup check always passes and the poll re-queues forever. Work never reaches the agent; dashboard shows a stuck/no-op task.

Exact cause

Commit 44f914c bumped github.com/samcharles93/NellDB v0.2.5 -> v0.3.0 touching only go.mod/go.sum. v0.3.0 changed the DocDB rev-cache rebuild:

  • v0.2.5 sdk/docdb.go reindex() indexed EVERY record by rec.ID, including Archie's meta:task_id_counter / meta:event_id_counter docs.
  • v0.3.0 sdk/docdb.go reindex() skips any id where isInternalID() is true (sdk/replicate.go:841: id[:5] == "meta:"), intended only for the SDK's own meta:clock/meta:vector bookkeeping but prefix-matching Archie's counter docs too.
  • v0.3.0 putIn (sdk/docdb.go:208-216) keeps the same conflict rule as v0.2.5: incomingRev != "" && !exists -> sdk.ErrConflict.

Archie's Adapter.nextCounter (internal/nell/adapter.go:957) is a Get -> mutate -> Put RMW that supplies the _rev returned by Get (joinDoc copies the payload, which putIn re-stamps with _rev). After any restart the counter docs are present in the log but absent from the rev cache, so exists=false and every counter Put conflicts. The counter doc can never be written again, so the cache is never repopulated — the failure is permanent for the lifetime of the DB, every process run, from the first poll.

Repro (standalone, module cache, v0.3.0): open log, Put meta:task_id_counter, close, reopen, Get (rev e.g. "1-69bfa51d..."), Put with that rev -> "sdk: revision conflict". First run of a fresh DB succeeds (empty-rev init Put is allowed); every subsequent process restart fails.

Why tests are green: OpenMemory/fresh-log tests initialize the counter via empty-rev Put within the same process; TestPersistenceAcrossRestart reopens a real log but only reads (TaskByIssue) and never exercises the counter RMW after reopen. The production path (restart an existing log, then enqueue) is untested.

Evidence

  • go.mod: github.com/samcharles93/NellDB v0.3.0 (no replace).
  • git show 44f914c --stat: go.mod, go.sum, .gitignore only; internal/nell/ untouched since.
  • Module cache diff !nell!d!b@v0.2.5 vs @v0.3.0: sdk/docdb.go reindex()/putIn(), sdk/replicate.go isInternalID.
  • /tmp/nell-repro: deterministic ErrConflict on counter RMW after reopen.

Fix direction

Options, in order of preference: (1) pin back to v0.2.5 (verified-2026-07-28 semantics) until adapter is migrated; (2) stop supplying _rev in nextCounter (drop rev from the RMW doc before Put — v0.3.0 continues the chain from the cached rev when incomingRev == ""), but that breaks the MVCC chain for the counter when the cache lacks it; (3) upstream: isInternalID should match exact SDK ids (meta:clock, meta:vector), not the meta: prefix, or reindex should index meta docs. Any fix must add a red test: open persisted log, EnqueueIssue, reopen, EnqueueIssue again -> no conflict.


Imported from beads archie-core-1786637491729-74-ee80f814 · GitHub #95 · type bug · priority P2 · created 2026-08-08 · closed 2026-08-08

## Symptom (observed 2026-08-05, task dev session) - Every 1m poll logs `issue queued` for the same open issue (sam/archie-core#8) again. - Every attempt logs `event sink insert failed: err="sdk: revision conflict"` (cmd/archied/main.go:373, the event-sink goroutine calling st.InsertEvent). - The daemon's own task consumer logs `nats enqueue failed: err="sdk: revision conflict"` (internal/daemon/daemon.go:599, calling Store.EnqueueIssue) and NAKs; JetStream redelivers to max-deliver 3 (triples of errors every ~2 min in the log). - The task doc is never persisted, so the enqueue dedup check always passes and the poll re-queues forever. Work never reaches the agent; dashboard shows a stuck/no-op task. ## Exact cause Commit 44f914c bumped github.com/samcharles93/NellDB v0.2.5 -> v0.3.0 touching only go.mod/go.sum. v0.3.0 changed the DocDB rev-cache rebuild: - v0.2.5 sdk/docdb.go reindex() indexed EVERY record by rec.ID, including Archie's meta:task_id_counter / meta:event_id_counter docs. - v0.3.0 sdk/docdb.go reindex() skips any id where isInternalID() is true (sdk/replicate.go:841: id[:5] == "meta:"), intended only for the SDK's own meta:clock/meta:vector bookkeeping but prefix-matching Archie's counter docs too. - v0.3.0 putIn (sdk/docdb.go:208-216) keeps the same conflict rule as v0.2.5: incomingRev != "" && !exists -> sdk.ErrConflict. Archie's Adapter.nextCounter (internal/nell/adapter.go:957) is a Get -> mutate -> Put RMW that supplies the _rev returned by Get (joinDoc copies the payload, which putIn re-stamps with _rev). After any restart the counter docs are present in the log but absent from the rev cache, so exists=false and every counter Put conflicts. The counter doc can never be written again, so the cache is never repopulated — the failure is permanent for the lifetime of the DB, every process run, from the first poll. Repro (standalone, module cache, v0.3.0): open log, Put meta:task_id_counter, close, reopen, Get (rev e.g. "1-69bfa51d..."), Put with that rev -> "sdk: revision conflict". First run of a fresh DB succeeds (empty-rev init Put is allowed); every subsequent process restart fails. Why tests are green: OpenMemory/fresh-log tests initialize the counter via empty-rev Put within the same process; TestPersistenceAcrossRestart reopens a real log but only reads (TaskByIssue) and never exercises the counter RMW after reopen. The production path (restart an existing log, then enqueue) is untested. ## Evidence - go.mod: github.com/samcharles93/NellDB v0.3.0 (no replace). - git show 44f914c --stat: go.mod, go.sum, .gitignore only; internal/nell/ untouched since. - Module cache diff !nell!d!b@v0.2.5 vs @v0.3.0: sdk/docdb.go reindex()/putIn(), sdk/replicate.go isInternalID. - /tmp/nell-repro: deterministic ErrConflict on counter RMW after reopen. ## Fix direction Options, in order of preference: (1) pin back to v0.2.5 (verified-2026-07-28 semantics) until adapter is migrated; (2) stop supplying _rev in nextCounter (drop rev from the RMW doc before Put — v0.3.0 continues the chain from the cached rev when incomingRev == ""), but that breaks the MVCC chain for the counter when the cache lacks it; (3) upstream: isInternalID should match exact SDK ids (meta:clock, meta:vector), not the meta: prefix, or reindex should index meta docs. Any fix must add a red test: open persisted log, EnqueueIssue, reopen, EnqueueIssue again -> no conflict. --- _Imported from beads `archie-core-1786637491729-74-ee80f814` · GitHub #95 · type bug · priority P2 · created 2026-08-08 · closed 2026-08-08_
sam added the mediumbug labels 2026-10-27 13:36:59 +00:00
sam self-assigned this 2026-10-27 13:36:59 +00:00
sam closed this issue 2026-10-27 13:36:59 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#602