Files
archie/docs/prds/payload-field-mapping.md
T
sam 9864766804 docs: describe the PostgreSQL-only stores, state_dir and the import-only db_path
Architecture, deployment, installer, skill and PRD text now name the Postgres
State Store, the offline recovery commands on database_url, state_dir for host
state, and db_path as the legacy-file locator for archie-state-store import.
Finalised PRDs, CHANGELOG, docs/news and docs/archive are unchanged records.
2026-09-24 14:51:48 +10:00

7.3 KiB

Payload field mapping -- decision

Status: Approved Date: 2026-08-22 Beads issue: archie-core-t2db.3, depends on t2db.1, blocks t2db.4

Answers what a "mapping" is, where it lives, and what "resolve against a real event" means precisely enough to implement. t2db.4 (binding: matcher

  • mapping + workflow) is out of scope -- this bead ships the mapping concept and its preview only, as a standalone entity t2db.4 will later reference by ID.

Decision: a new internal/domain/mapping package, SQLite storage

Storage superseded: every store now lives in PostgreSQL behind internal/domain/storecontract, implemented in internal/infrastructure/postgres (docs/prds/state-store-contract.md). SQLite names below describe the original design.

internal/domain/mapping owns the mapping vocabulary and the pure resolve/preview rule; internal/store persists it (mirrors captures.go's shape exactly -- inline schema const, narrow store interface, *store.Store implementation); internal/webui/api_mapping.go is the transport. This follows docs/prds/event-capture-storage.md's own placement precedent: internal/store is where this daemon's persistence already lives, not internal/infrastructure (that layer holds cross-domain services -- config, eventbus, modelcatalog -- not this daemon's primary store).

Why a domain package and not just a store table

Resolving a path against a payload and deciding "missing vs null vs type-changed" is a rule, not a query -- CLAUDE.md's domain layer owns "language, state, rules, operations" precisely so this logic is testable without a database and reusable by t2db.4's runtime trigger path without either package importing the other's transport concerns.

Mapping shape

type Mapping struct {
	ID         int64
	Name       string
	SourceHint string   // free-text label (e.g. "sentry"), organisational only -- not enforced against Capture.Source
	Fields     []Field
	CreatedAt  time.Time
	UpdatedAt  time.Time
}

type Field struct {
	Name     string    // the name the workflow receives
	Path     string    // dot/bracket path into the payload, see below
	Type     FieldType // expected type, checked at resolve time
	Required bool      // Required=false + missing is not a failure; Required=false + wrong type still is
}

type FieldType string // "string" | "number" | "bool" | "object" | "array" | "any"

SourceHint is advisory only: an operator authors a mapping by clicking through one example capture, but the mapping is not pinned to that capture's exact source string, because t2db.4's matcher (not this bead) decides which events a binding applies to. Pinning here would duplicate that decision in the wrong layer.

Path syntax

Dot-separated keys with [N] for array indices: data.items[0].id. No wildcards, no filters, no external library -- the grammar is exactly what a click-through UI needs to express (this key, then this key, then this index), and every path in a mapping was generated by clicking a value in a real payload, never hand-written against a spec. A hand-rolled parser (~30 lines: split on ., strip [N] suffixes) is less risk than a new dependency for a grammar this small.

Resolve semantics (the preview AND the future runtime path)

func Resolve(fields []Field, payload []byte) (values map[string]any, failures []Failure)

type Failure struct {
	FieldName string
	Path      string
	Reason    string // "missing" | "null" | "type_mismatch"
}

Decode with json.Decoder.UseNumber() (matches webhookguard.RedactPayload's existing convention) so int/float distinction survives for type checking. Walk each field's path independently -- one field's failure does not stop resolution of the others, because the whole point of preview is showing the operator every problem at once, not the first one.

  • Path segment doesn't exist, or indexes past an array's length -> missing
  • Path resolves to JSON null -> null
  • Path resolves to a value whose JSON type doesn't match Field.Type ("any" always matches) -> type_mismatch
  • Required: false suppresses missing only, never null or type_mismatch -- a field that's present but the wrong shape is always worth surfacing, since silently coercing it is exactly the "agent confabulates around a blank field" failure this bead exists to prevent.

This is the same function both consumers use. t2db.3's preview calls it against an operator-picked capture and renders every result, pass or fail, in the UI -- that satisfies this bead's "fail loudly" acceptance criterion, because at this stage the only runtime consumer of a mapping is the human previewing it. t2db.4 will call the identical function when a live event triggers a binding and turn any non-empty failures into a visible operator-facing failure (park the task / surface in the inspector) instead of starting a workflow with blank fields -- that wiring is t2db.4's job, not duplicated here.

Storage

The mappings table: id, name, source_hint, fields_json, created_at, updated_at. fields_json is the []Field slice JSON-encoded into one column -- fields have no independent query need (nothing filters "mappings with a field named X"), so a join table would be structure with no consumer, the same failure mode organisation.md bans for empty layering.

MappingStore interface (narrow, mirrors CaptureStore): InsertMapping, ListMappings, GetMapping, UpdateMapping, DeleteMapping.

HTTP surface

All under /api/mappings, token-gated like every other /api/* route (no bypass -- unlike capture, nothing here accepts unauthenticated input):

  • GET /api/mappings / POST /api/mappings -- list / create
  • GET|PATCH|DELETE /api/mappings/{id}
  • POST /api/mappings/preview -- body {fields: []Field, capture_id: int64}, calls mapping.Resolve against that capture's stored (already redacted) Body and returns {values, failures}. Deliberately takes loose fields rather than a saved mapping ID, so an operator can iterate before saving anything -- "never ship a mapping the operator has not seen resolve against real data" requires preview to work pre-save.

Preview resolves against CapturedEvent.Body, which is already redaction-passed (webhookguard.RedactPayload, per event-capture-storage.md) -- a mapped field pointing at a sensitive-keyed path previews as the literal string "[redacted]", not the real value. This is accepted, not fixed here: it is the stored ground truth, the same payload t2db.4's runtime path will resolve against, and a preview mismatch would be worse than an honest one.

Frontend

New ui/src/mappings/ feature folder, same three-file split as ui/src/captures/: mappings.js (page: list, fetch, create/edit flow), mapping-editor.js (pure, unit-testable: renders a captured payload as a clickable tree -- clicking a leaf appends a field row with Path pre-filled -- plus the field table and preview-result rendering), mappings.css. Creating a mapping starts from an existing captured event (reuses api.captures), so the editor always has a real payload to click through and preview against, never a blank form.

What this does not decide

The binding entity, its matcher, its draft/pending_approval/armed state machine, and turning a runtime Resolve failure into a parked task are t2db.4. Per-source HMAC secret registration is t2db.5. This bead ships the mapping concept, its storage, and its preview only.