Adds t2db.3: schema-by-example field mapping for the no-code playbook epic. An operator picks a captured event, clicks through its payload to bind named fields to JSON paths, and previews the resolution against a real event before saving -- a missing, null, or type-changed path fails loudly (named by field and event) rather than handing an agent a blank value. - internal/domain/mapping: Mapping/Field types, path parsing/walking, Resolve (per-field failure reporting), Validate (incl. duplicate names) - internal/store/mappings.go: SQLite persistence, mirrors captures.go - internal/webui/api_mapping.go: CRUD + preview HTTP API - ui/src/mappings/: dashboard editor (clickable payload tree, live preview) Design decisions recorded in docs/prds/payload-field-mapping.md. t2db.4 (binding a mapping + matcher to a workflow) is out of scope here.
160 lines
7.2 KiB
Markdown
160 lines
7.2 KiB
Markdown
# Payload field mapping -- decision
|
|
|
|
**Status:** Decided, not yet implemented
|
|
**Date:** 2026-08-22
|
|
**Beads issue:** `archie-core-t2db.3`, depends on `t2db.1`, blocks `t2db.4`
|
|
|
|
Answers what a "mapping" is, where it lives, and what "resolve against a
|
|
real event" means precisely enough to implement. `t2db.4` (binding: matcher
|
|
+ mapping + workflow) is out of scope -- this bead ships the mapping
|
|
concept and its preview only, as a standalone entity `t2db.4` will later
|
|
reference by ID.
|
|
|
|
## Decision: a new `internal/domain/mapping` package, SQLite storage
|
|
|
|
`internal/domain/mapping` owns the mapping vocabulary and the pure
|
|
resolve/preview rule; `internal/store` persists it (mirrors
|
|
`captures.go`'s shape exactly -- inline schema const, narrow store
|
|
interface, `*store.Store` implementation); `internal/webui/api_mapping.go`
|
|
is the transport. This follows `docs/prds/event-capture-storage.md`'s own
|
|
placement precedent: `internal/store` is where this daemon's persistence
|
|
already lives, not `internal/infrastructure` (that layer today holds
|
|
cross-domain services -- config, eventbus, modelcatalog -- not this
|
|
daemon's primary store).
|
|
|
|
### Why a domain package and not just a store table
|
|
|
|
Resolving a path against a payload and deciding "missing vs null vs
|
|
type-changed" is a rule, not a query -- CLAUDE.md's domain layer owns
|
|
"language, state, rules, operations" precisely so this logic is testable
|
|
without a database and reusable by `t2db.4`'s runtime trigger path without
|
|
either package importing the other's transport concerns.
|
|
|
|
## Mapping shape
|
|
|
|
```go
|
|
type Mapping struct {
|
|
ID int64
|
|
Name string
|
|
SourceHint string // free-text label (e.g. "sentry"), organisational only -- not enforced against Capture.Source
|
|
Fields []Field
|
|
CreatedAt time.Time
|
|
UpdatedAt time.Time
|
|
}
|
|
|
|
type Field struct {
|
|
Name string // the name the workflow receives
|
|
Path string // dot/bracket path into the payload, see below
|
|
Type FieldType // expected type, checked at resolve time
|
|
Required bool // Required=false + missing is not a failure; Required=false + wrong type still is
|
|
}
|
|
|
|
type FieldType string // "string" | "number" | "bool" | "object" | "array" | "any"
|
|
```
|
|
|
|
`SourceHint` is advisory only: an operator authors a mapping by clicking
|
|
through one example capture, but the mapping is not pinned to that
|
|
capture's exact source string, because `t2db.4`'s matcher (not this bead)
|
|
decides which events a binding applies to. Pinning here would duplicate
|
|
that decision in the wrong layer.
|
|
|
|
## Path syntax
|
|
|
|
Dot-separated keys with `[N]` for array indices: `data.items[0].id`. No
|
|
wildcards, no filters, no external library -- the grammar is exactly what a
|
|
click-through UI needs to express (this key, then this key, then this
|
|
index), and every path in a mapping was generated by clicking a value in a
|
|
real payload, never hand-written against a spec. A hand-rolled parser
|
|
(~30 lines: split on `.`, strip `[N]` suffixes) is less risk than a new
|
|
dependency for a grammar this small.
|
|
|
|
## Resolve semantics (the preview AND the future runtime path)
|
|
|
|
```go
|
|
func Resolve(fields []Field, payload []byte) (values map[string]any, failures []Failure)
|
|
|
|
type Failure struct {
|
|
FieldName string
|
|
Path string
|
|
Reason string // "missing" | "null" | "type_mismatch"
|
|
}
|
|
```
|
|
|
|
Decode with `json.Decoder.UseNumber()` (matches `webhookguard.RedactPayload`'s
|
|
existing convention) so int/float distinction survives for type checking.
|
|
Walk each field's path independently -- one field's failure does not stop
|
|
resolution of the others, because the whole point of preview is showing
|
|
the operator every problem at once, not the first one.
|
|
|
|
- Path segment doesn't exist, or indexes past an array's length -> `missing`
|
|
- Path resolves to JSON `null` -> `null`
|
|
- Path resolves to a value whose JSON type doesn't match `Field.Type`
|
|
(`"any"` always matches) -> `type_mismatch`
|
|
- `Required: false` suppresses `missing` only, never `null` or
|
|
`type_mismatch` -- a field that's present but the wrong shape is always
|
|
worth surfacing, since silently coercing it is exactly the "agent
|
|
confabulates around a blank field" failure this bead exists to prevent.
|
|
|
|
**This is the same function both consumers use.** `t2db.3`'s preview calls
|
|
it against an operator-picked capture and renders every result, pass or
|
|
fail, in the UI -- that satisfies this bead's "fail loudly" acceptance
|
|
criterion, because at this stage the only runtime consumer of a mapping
|
|
*is* the human previewing it. `t2db.4` will call the identical function
|
|
when a live event triggers a binding and turn any non-empty `failures` into
|
|
a visible operator-facing failure (park the task / surface in the
|
|
inspector) instead of starting a workflow with blank fields -- that wiring
|
|
is `t2db.4`'s job, not duplicated here.
|
|
|
|
## Storage
|
|
|
|
`internal/store/mappings.go`, `field_mappings` table: `id, name,
|
|
source_hint, fields_json, created_at, updated_at`. `fields_json` is the
|
|
`[]Field` slice JSON-encoded into one column -- fields have no independent
|
|
query need (nothing filters "mappings with a field named X" today), so a
|
|
join table would be structure with no consumer, the same failure mode
|
|
`organisation.md` bans for empty layering.
|
|
|
|
`MappingStore` interface (narrow, mirrors `CaptureStore`):
|
|
`InsertMapping`, `ListMappings`, `GetMapping`, `UpdateMapping`,
|
|
`DeleteMapping`.
|
|
|
|
## HTTP surface
|
|
|
|
All under `/api/mappings`, token-gated like every other `/api/*` route (no
|
|
bypass -- unlike capture, nothing here accepts unauthenticated input):
|
|
|
|
- `GET /api/mappings` / `POST /api/mappings` -- list / create
|
|
- `GET|PATCH|DELETE /api/mappings/{id}`
|
|
- `POST /api/mappings/preview` -- body `{fields: []Field, capture_id:
|
|
int64}`, calls `mapping.Resolve` against that capture's stored (already
|
|
redacted) `Body` and returns `{values, failures}`. Deliberately takes
|
|
loose `fields` rather than a saved mapping ID, so an operator can iterate
|
|
before saving anything -- "never ship a mapping the operator has not seen
|
|
resolve against real data" requires preview to work pre-save.
|
|
|
|
Preview resolves against `CapturedEvent.Body`, which is already
|
|
redaction-passed (`webhookguard.RedactPayload`, per
|
|
`event-capture-storage.md`) -- a mapped field pointing at a
|
|
sensitive-keyed path previews as the literal string `"[redacted]"`, not the
|
|
real value. This is accepted, not fixed here: it is the stored ground
|
|
truth, the same payload `t2db.4`'s runtime path will resolve against, and a
|
|
preview mismatch would be worse than an honest one.
|
|
|
|
## Frontend
|
|
|
|
New `ui/src/mappings/` feature folder, same three-file split as
|
|
`ui/src/captures/`: `mappings.js` (page: list, fetch, create/edit flow),
|
|
`mapping-editor.js` (pure, unit-testable: renders a captured payload as a
|
|
clickable tree -- clicking a leaf appends a field row with `Path`
|
|
pre-filled -- plus the field table and preview-result rendering),
|
|
`mappings.css`. Creating a mapping starts from an existing captured event
|
|
(reuses `api.captures`), so the editor always has a real payload to click
|
|
through and preview against, never a blank form.
|
|
|
|
## What this does not decide
|
|
|
|
The binding entity, its matcher, its draft/pending_approval/armed state
|
|
machine, and turning a runtime `Resolve` failure into a parked task are
|
|
`t2db.4`. Per-source HMAC secret registration is `t2db.5`. This bead ships
|
|
the mapping concept, its storage, and its preview only.
|