diff options
Diffstat (limited to 'docs/superpowers')
| -rw-r--r-- | docs/superpowers/specs/2026-09-20-artifact-pipeline-design.md | 236 |
1 files changed, 236 insertions, 0 deletions
diff --git a/docs/superpowers/specs/2026-09-20-artifact-pipeline-design.md b/docs/superpowers/specs/2026-09-20-artifact-pipeline-design.md new file mode 100644 index 0000000..4fe314c --- /dev/null +++ b/docs/superpowers/specs/2026-09-20-artifact-pipeline-design.md @@ -0,0 +1,236 @@ +# Artifact Pipeline Design + +## Goal + +Turn the existing F# replay/server scaffold into a verifiable design-run +pipeline without replacing the existing UI or shared geometry model. A run +must produce immutable, addressable geometry artifacts, an atomic manifest, and +observable lifecycle state that the server and browser consume from the same +F# types. + +## Chosen Architecture + +`SomhairlesDream.Shared` owns the wire and persistence contract. One +`SomhairlesDream.Modeling` F# library owns design definitions, artifact +writing, the Blender bridge, and pipeline orchestration. Add +`SomhairlesDream.Cli` as a thin executable over that library. The server +references the library and starts the same pipeline in the background. This +keeps the CLI and HTTP path behaviorally identical while keeping Blender +execution out of the browser and out of production source files. + +The Blender stage is isolated under `stage/blender/fixture.py`. It is a fixed, +small fixture, not a production model loader: it accepts only a known step ID, +builds deterministic primitives, requires `bpy.app.version >= 3.0.0`, exports +GLB, reimports the GLB to verify object IDs, and optionally renders PNG. The +F# bridge invokes a configured Python executable that imports `bpy`; it never +evaluates user-supplied code or shell strings. + +## Shared Contract + +`SomhairlesDream.Shared` owns these records and lifecycle types. Resource IDs +(`ProjectId`, `RunId`, `BaseId`, and `TargetId`) are ASCII strings matching +`[a-z0-9][a-z0-9-]{0,63}` and are validated before any filesystem path is +built. Object IDs use the separate pattern `[a-z0-9][a-z0-9.-]{0,63}` so +namespaced IDs such as `heritage.foundation` are valid. + +- `RunIds`: `ProjectId`, `RunId`, `BaseId`, and `TargetId`. +- `RunStart`: IDs, start timestamp, and total step count. +- `RunCheckpoint`: IDs, checkpoint timestamp, step ID/index, total steps, + artifact path, and message. +- `RunHeartbeat`: IDs, timestamp, and message. +- `RunComplete`: IDs, completion timestamp, and manifest path. +- `RunFail`: IDs, failure timestamp, optional step ID, and message. +- `RunEvent`: the `Start`, `Checkpoint`, `Heartbeat`, `Complete`, and `Fail` + union consumed by the pipeline, CLI event log, and server store. The NDJSON + encoding is a flat object with `kind`, `at`, the four IDs, optional `stepId`, + optional `stepIndex`, optional `totalSteps`, optional `message`, optional + `artifactPath`, and optional `manifestPath`; `kind` is one of `start`, + `checkpoint`, `heartbeat`, `complete`, or `fail`. +- `ArtifactChange`: object ID, change kind, and human-readable summary. +- `RenderArtifact`: relative PNG path, SHA-256, byte count, and export time. +- `StepArtifact`: step metadata, immutable relative GLB path, SHA-256, byte + count, actual Blender export timestamp, source object IDs, verified object + IDs from GLB reimport, changes, and optional render. +- `ArtifactManifest`: schema version, run IDs, lifecycle timestamps, Blender + version, ordered step artifacts, and final status. + +The manifest JSON uses camelCase names and ISO-8601 UTC timestamps. Every step +has a unique ID and path below its run directory. Object IDs are stable +strings such as `heritage.foundation` and are written both as Blender object +names and under the custom-property key `object_id`. The fixture checks the +source custom property before export; GLB reimport verifies object names and +records those names as `verifiedObjectIds`, because object names are the +portable GLB identity field. `changes` describes the intended delta for that +step, while the GLB is a complete snapshot of the step. + +Every event includes a non-null UTC `at` timestamp. `start` requires all four +IDs and `totalSteps`; `checkpoint` requires `stepId`, zero-based `stepIndex`, +`totalSteps`, `artifactPath`, and `message`; `heartbeat` requires a message; `complete` +requires `manifestPath`; and `fail` requires a message and may include +`stepId`. Fields that do not apply are omitted rather than encoded as `null`. +The serialized `stepIndex` is zero-based and the human-facing checkpoint label +is one-based only in UI text. + +The fixed design contains these ordered steps: + +| Step | Object IDs in the complete snapshot | Changes | +| --- | --- | --- | +| `01-foundation` | `heritage.foundation`, `heritage.deck` | Add foundation slab and deck | +| `02-frame` | Foundation IDs plus `heritage.frame.left`, `heritage.frame.right`, `heritage.spine` | Add the primary frame and spine | +| `03-cabin` | Frame IDs plus `heritage.cabin`, `heritage.crossbeam` | Add cabin volume and crossbeam | + +The default run IDs are `heritage-001`, `run-001`, `base-000`, and +`target-003`; CLI and HTTP callers may provide other validated IDs. + +## Filesystem Contract + +Each run is isolated at: + +```text +.artifacts/<projectId>/<runId>/ + events.ndjson + manifest.json + steps/ + 01-foundation.glb + 02-frame.glb + 03-cabin.glb + 01-foundation.png # optional +``` + +GLB and render files are written to a per-step staging directory, hashed after +the Blender process exits successfully, and moved into their final names with +no-overwrite semantics. Any pre-existing run directory, including one with +only `events.ndjson` or abandoned temporary files, is rejected; reruns use a +new `RunId`. Staging is cleaned on failure, while already-created immutable +final steps are retained for forensic inspection and are never described by a +manifest. `manifest.json` is written to a sibling temporary file, flushed, +then atomically renamed into place. The manifest rename is the irreversible +commit point: before it, any error emits `Fail` and no manifest is present; +after it, hash and path verification have already passed and the manifest's +status is `complete`. The `Complete` event is appended after the rename; a +crash or event-log failure at that point leaves the manifest authoritative and +is recovered as complete on the next inspection, without deleting or +rewriting it. A manifest is never published if any required step fails before +the commit point. `verify` recomputes all +recorded hashes, checks paths stay inside the run directory, checks the source +and verified object ID sets recorded by the bridge, checks timestamps, and +rejects duplicate step paths. Reimport is performed by the real stage +integration command before the manifest is written; `verify` does not need +Blender and validates the recorded proof plus the design's expected ID set. + +## Lifecycle and Failure Behavior + +The transition rules are strict: `Start` is accepted only from `Idle`, +`Checkpoint` and `Heartbeat` only from `Running`, `Complete` only after all +ordered steps are checkpointed, and `Fail` from any nonterminal state. +Terminal states reject duplicate completion or failure. The pipeline emits +`Start`, then a heartbeat immediately and every 5 seconds while each Blender +process is alive, one checkpoint after each immutable artifact is verified, and +`Complete` only after the atomic manifest rename. Any bridge, hash, validation, +or filesystem failure before the commit point emits `Fail` with the current +step and leaves no final manifest. The server maps the same events into SSE +snapshots. A run is stale +when its status is `running` and `now - lastHeartbeatAt > 15 seconds`; a +one-second server monitor recalculates that derived flag and emits a snapshot +even when no lifecycle event arrives. Staleness does not silently complete or +fail the run. The existing replay UI remains available independently of +artifact execution. + +The bridge protocol is fixed. The F# process invokes: + +```text +<python> stage/blender/fixture.py --step-id <id> --output <tmp.glb> + [--render-output <tmp.png>] +``` + +`<python>` comes from `--python` or `SOMHAIRLES_BPYTHON`; the default is +`python3`. The script must import `bpy` and report version `>= 3.0.0`. Exit 0 +prints exactly one JSON result object containing `ok`, `blenderVersion`, +`exportedAt`, `objectIds`, `verifiedObjectIds`, `glbBytes`, and optional +`renderPath`/`renderedAt`. Exit 2 means invalid step or arguments, exit 3 +means missing/unsupported `bpy`, and exit 4 means export or render failure. +The bridge timeout is 120 seconds per step. `exportedAt` is emitted by Python +immediately after the successful GLB export; F# rejects a missing or non-UTC +timestamp. `verifiedObjectIds` comes from reimporting the just-written GLB and +must equal the design's expected IDs. + +## CLI and Server Surface + +The CLI exposes: + +```text +dotnet run --project src/SomhairlesDream.Cli -- run [options] +dotnet run --project src/SomhairlesDream.Cli -- verify <manifest.json> +``` + +`run` accepts `--project-id`, `--run-id`, `--base-id`, `--target-id`, +`--output-root`, `--stage-root`, `--python`, and `--render`. Defaults are the +IDs above, `.artifacts`, `stage`, `SOMHAIRLES_BPYTHON`/`python3`, and no +render. It writes `events.ndjson` and prints the final manifest path. Exit 0 +means complete, 2 means invalid arguments, 3 means bridge failure, 4 means +artifact/manifest failure. `verify` exits 0 only when every hash and contract +check passes, and exits 4 otherwise. + +`RunSnapshot` is also defined in `SomhairlesDream.Shared` and is the single +JSON/SSE projection used by the server and browser. The server accepts JSON +bodies with the four IDs and optional `render` on +`POST /api/runs/start`; it returns HTTP 202 with the accepted `RunSnapshot` in +the response body, or 409 for an already-running `(ProjectId, RunId)`. Invalid +JSON or IDs returns 400; +unknown manifest/run selectors return 404; a terminal run returns 409 for a +second start. It exposes +`GET /api/runs/events?projectId=...&runId=...` for that exact run and +`GET /api/artifacts/manifest?projectId=...&runId=...` for its completed +manifest; that endpoint returns 200 with the manifest after completion and 409 +while the selected run is still active. The start endpoint launches the same +executor asynchronously. Checkpoint, heartbeat, complete, and fail are +internal pipeline/store calls, not public HTTP commands, so callers cannot +bypass artifact validation to force a terminal state. Only one run with a +given project/run pair may be active; different run IDs may execute +independently. + +The SSE `RunSnapshot` schema is explicit: `projectId`, `runId`, `baseId`, +`targetId`, `status` (`idle|running|complete|failed`), optional +`currentStepId`, optional `stepIndex`, `totalSteps`, optional +`lastHeartbeatAt`, `isStale`, optional `error`, and optional `manifestUrl`. +The query parameters prevent event multiplexing between runs. + +On successful completion the browser's telemetry panel displays the current +step and a link with ID `artifact-manifest-link` to the manifest endpoint. A +failed or stale run displays the error state and no link. + +Code publication is separate from runtime artifact publication. This task has +explicit authorization to publish after all verification passes. The scoped +commit set is the project code, tests, stage fixture, and design documentation +on `main`; no unrelated worktree files are staged. Create only +`/home/somhairle/git/somhairles-dream-fsharp.git` with `git init --bare`, add +local-path origin `/home/somhairle/git/somhairles-dream-fsharp.git`, and push +`main` without force. Verify `git ls-remote` contains the pushed ref and +verify the public cgit tree at +`https://git.somhairle.bid/somhairles-dream-fsharp/` (or the URL discovered +from the cgit response). If that path already exists, do not reinitialize or +overwrite it: inspect that it is a bare repository with the expected project +ref, otherwise stop and report the exact boundary. No Docker, nginx, cgit +configuration, auth files, or other repositories are changed. + +## Verification + +- Shared tests cover IDs, lifecycle transitions, manifest shape, and path/hash + validation. +- Modeling tests use a fake bridge for deterministic atomic-writer tests. +- A stage integration command runs the real `bpy 5.0.1` fixture, verifies all + three GLBs and optional renders, then runs CLI `verify`. +- Server verification checks `/health`, lifecycle JSON/SSE, and manifest + serving. +- Browser verification checks SSE reaches completion, exposes the manifest + link, and preserves the already-fixed `Uint32BufferAttribute` mesh path. + +## Alternatives Considered + +1. **CLI-only pipeline:** smallest server change, but HTTP and CLI behavior + could diverge and the browser could not observe real run progress. +2. **Server-owned Blender process:** direct integration, but it couples the + web host to stage tooling and makes CLI verification harder. +3. **Shared library plus thin CLI and server adapter (chosen):** one typed + pipeline implementation, isolated Blender fixture, and both local and HTTP + execution paths with explicit boundaries. |
