v1.20.0 archied startup panic and schema-unsafe update rollback #1324

Closed
opened 2026-10-27 13:39:56 +00:00 by sam · 1 comment
Owner

Bug Description

After /update installed archied v1.20.0, the service failed to start. The update watchdog reported health_check: failed and rolled the binary back to v1.19.10, but the v1.20.0 binary had already migrated the task database to schema version 2. The rollback binary then entered a restart loop because it only supports schema version 1.

Restoring the matching v1.20.0 daemon did not recover the service: it panics during startup in (*boot).connectNATS at bootstrap.go:366 while dereferencing cfg.NATS.URL.

Steps to Reproduce

  1. Run /update from the Telegram gateway, selecting archied v1.20.0.
  2. Observe the watchdog health check fail and rollback.
  3. Start the matching v1.20.0 binary against the existing config.

Expected Behavior

The released daemon starts successfully with the production config, or returns a clear configuration error. /update must not leave the installed binary and persisted task schema at incompatible versions.

Actual Behavior

The service is in a systemd restart loop. v1.19.10 reports store: migrate: database schema version 2 is newer than supported version 1; v1.20.0 panics with runtime error: invalid memory address or nil pointer dereference in connectNATS at bootstrap.go:366.

Evidence

  • Installed before recovery: archied 1.19.10, archie-agent 1.19.9
  • Update report: installed daemon/agent 1.20.0, health_check: failed, rolled_back: true
  • Task DB: /home/sam/.local/share/archie/archie.db-tasks.sqlite, PRAGMA user_version = 2
  • v1.20.0 panic: url := cfg.NATS.URL in internal/app/archied/bootstrap.go:366
  • Current config contains [nats] url = "nats://172.19.0.1:4222"

Acceptance Criteria

  • Add a regression test covering startup with the production NATS configuration and nil/invalid NATS state.
  • Remove the nil dereference; startup must return a useful error or initialise the required NATS configuration.
  • Make update rollback schema-safe: either verify the candidate before migration, retain/restore compatible state, or prevent rollback to a binary that cannot read the candidate schema.
  • Exercise /update or an equivalent installer/restart path and verify archied.service reaches active, Telegram gateway starts, NATS connects, and /version reports the installed release.
  • Preserve the task database and existing tasks.

Environment

  • Host: carina.srv.catlow.cloud
  • OS: Debian/Linux 6.12.96+deb13-amd64
  • Service: systemd user unit archied.service
  • Update path: Telegram /update

Close reason

Resolved; GitHub #703 closed. Stale-open copy recovered from carina during machine sync.


Imported from beads archie-core-fwvf · GitHub #703 · type bug · priority P1 · created 2026-09-03 · closed 2026-09-03

## Bug Description After `/update` installed `archied` v1.20.0, the service failed to start. The update watchdog reported `health_check: failed` and rolled the binary back to v1.19.10, but the v1.20.0 binary had already migrated the task database to schema version 2. The rollback binary then entered a restart loop because it only supports schema version 1. Restoring the matching v1.20.0 daemon did not recover the service: it panics during startup in `(*boot).connectNATS` at `bootstrap.go:366` while dereferencing `cfg.NATS.URL`. ## Steps to Reproduce 1. Run `/update` from the Telegram gateway, selecting archied v1.20.0. 2. Observe the watchdog health check fail and rollback. 3. Start the matching v1.20.0 binary against the existing config. ## Expected Behavior The released daemon starts successfully with the production config, or returns a clear configuration error. `/update` must not leave the installed binary and persisted task schema at incompatible versions. ## Actual Behavior The service is in a systemd restart loop. v1.19.10 reports `store: migrate: database schema version 2 is newer than supported version 1`; v1.20.0 panics with `runtime error: invalid memory address or nil pointer dereference` in `connectNATS` at `bootstrap.go:366`. ## Evidence - Installed before recovery: `archied 1.19.10`, `archie-agent 1.19.9` - Update report: installed daemon/agent `1.20.0`, `health_check: failed`, `rolled_back: true` - Task DB: `/home/sam/.local/share/archie/archie.db-tasks.sqlite`, `PRAGMA user_version = 2` - v1.20.0 panic: `url := cfg.NATS.URL` in `internal/app/archied/bootstrap.go:366` - Current config contains `[nats] url = "nats://172.19.0.1:4222"` ## Acceptance Criteria - Add a regression test covering startup with the production NATS configuration and nil/invalid NATS state. - Remove the nil dereference; startup must return a useful error or initialise the required NATS configuration. - Make update rollback schema-safe: either verify the candidate before migration, retain/restore compatible state, or prevent rollback to a binary that cannot read the candidate schema. - Exercise `/update` or an equivalent installer/restart path and verify `archied.service` reaches active, Telegram gateway starts, NATS connects, and `/version` reports the installed release. - Preserve the task database and existing tasks. ## Environment - Host: carina.srv.catlow.cloud - OS: Debian/Linux 6.12.96+deb13-amd64 - Service: systemd user unit `archied.service` - Update path: Telegram `/update` ### Close reason Resolved; GitHub #703 closed. Stale-open copy recovered from carina during machine sync. --- _Imported from beads `archie-core-fwvf` · GitHub #703 · type bug · priority P1 · created 2026-09-03 · closed 2026-09-03_
sam added the highbug labels 2026-10-27 13:39:56 +00:00
sam self-assigned this 2026-10-27 13:39:56 +00:00
sam closed this issue 2026-10-27 13:39:56 +00:00
Author
Owner

sam wrote (originally 2026-09-03):

Investigation correction

The panic frame is setupObservability, not connectNATS. In the released/latest source, setupObservability assigns b.d.Mappings = ms while b.d is still nil; buildDaemon runs later in startup. The direct nil dereference is therefore the daemon wiring order, while the schema-unsafe rollback remains the separate update failure.

The implementation and regression test will cover both paths.

_sam wrote (originally 2026-09-03):_ ## Investigation correction The panic frame is `setupObservability`, not `connectNATS`. In the released/latest source, `setupObservability` assigns `b.d.Mappings = ms` while `b.d` is still nil; `buildDaemon` runs later in startup. The direct nil dereference is therefore the daemon wiring order, while the schema-unsafe rollback remains the separate update failure. The implementation and regression test will cover both paths.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1324