path: root/docs/superpowers/specs/2026-09-20-artifact-pipeline-design.md

# Artifact Pipeline Design

## Goal

Turn the existing F# replay/server scaffold into a verifiable design-run
pipeline without replacing the existing UI or shared geometry model. A run
must produce immutable, addressable geometry artifacts, an atomic manifest, and
observable lifecycle state that the server and browser consume from the same
F# types.

## Chosen Architecture

`SomhairlesDream.Shared` owns the wire and persistence contract. One
`SomhairlesDream.Modeling` F# library owns design definitions, artifact
writing, the Blender bridge, and pipeline orchestration. Add
`SomhairlesDream.Cli` as a thin executable over that library. The server
references the library and starts the same pipeline in the background. This
keeps the CLI and HTTP path behaviorally identical while keeping Blender
execution out of the browser and out of production source files.

The Blender stage is isolated under `stage/blender/fixture.py`. It is a fixed,
small fixture, not a production model loader: it accepts only a known step ID,
builds deterministic primitives, requires `bpy.app.version >= 3.0.0`, exports
GLB, reimports the GLB to verify object IDs, and optionally renders PNG. The
F# bridge invokes a configured Python executable that imports `bpy`; it never
evaluates user-supplied code or shell strings.

## Shared Contract

`SomhairlesDream.Shared` owns these records and lifecycle types. Resource IDs
(`ProjectId`, `RunId`, `BaseId`, and `TargetId`) are ASCII strings matching
`[a-z0-9][a-z0-9-]{0,63}` and are validated before any filesystem path is
built. Object IDs use the separate pattern `[a-z0-9][a-z0-9.-]{0,63}` so
namespaced IDs such as `heritage.foundation` are valid.

- `RunIds`: `ProjectId`, `RunId`, `BaseId`, and `TargetId`.
- `RunStart`: IDs, start timestamp, and total step count.
- `RunCheckpoint`: IDs, checkpoint timestamp, step ID/index, total steps,
  artifact path, and message.
- `RunHeartbeat`: IDs, timestamp, and message.
- `RunComplete`: IDs, completion timestamp, and manifest path.
- `RunFail`: IDs, failure timestamp, optional step ID, and message.
- `RunEvent`: the `Start`, `Checkpoint`, `Heartbeat`, `Complete`, and `Fail`
  union consumed by the pipeline, CLI event log, and server store. The NDJSON
  encoding is a flat object with `kind`, `at`, the four IDs, optional `stepId`,
  optional `stepIndex`, optional `totalSteps`, optional `message`, optional
  `artifactPath`, and optional `manifestPath`; `kind` is one of `start`,
  `checkpoint`, `heartbeat`, `complete`, or `fail`.
- `ArtifactChange`: object ID, change kind, and human-readable summary.
- `RenderArtifact`: relative PNG path, SHA-256, byte count, and export time.
- `StepArtifact`: step metadata, immutable relative GLB path, retained process
  log path, SHA-256, byte count, actual Blender export timestamp, source object
  IDs, verified object IDs from GLB reimport, changes, and optional render.
- `ArtifactManifest`: schema version, run IDs, lifecycle timestamps, Blender
  version, ordered step artifacts, and final status.

The manifest JSON uses camelCase names and ISO-8601 UTC timestamps. Every step
has a unique ID and path below its run directory. Object IDs are stable
strings such as `heritage.foundation` and are written both as Blender object
names and under the custom-property key `object_id`. The fixture checks the
source custom property before export; GLB reimport verifies object names and
records those names as `verifiedObjectIds`, because object names are the
portable GLB identity field. `changes` describes the intended delta for that
step, while the GLB is a complete snapshot of the step.

Every event includes a non-null UTC `at` timestamp. `start` requires all four
IDs and `totalSteps`; `checkpoint` requires `stepId`, zero-based `stepIndex`,
`totalSteps`, `artifactPath`, and `message`; `heartbeat` requires a message; `complete`
requires `manifestPath`; and `fail` requires a message and may include
`stepId`. Fields that do not apply are omitted rather than encoded as `null`.
The serialized `stepIndex` is zero-based and the human-facing checkpoint label
is one-based only in UI text.

The fixed design contains these ordered steps:

| Step | Object IDs in the complete snapshot | Changes |
| --- | --- | --- |
| `01-foundation` | `heritage.foundation`, `heritage.deck` | Add foundation slab and deck |
| `02-frame` | Foundation IDs plus `heritage.frame.left`, `heritage.frame.right`, `heritage.spine` | Add the primary frame and spine |
| `03-cabin` | Frame IDs plus `heritage.cabin`, `heritage.crossbeam` | Add cabin volume and crossbeam |

The default run IDs are `heritage-001`, `run-001`, `base-000`, and
`target-003`; CLI and HTTP callers may provide other validated IDs.

## Filesystem Contract

Each run is isolated at:

```text
.artifacts/<projectId>/<runId>/
  events.ndjson
  manifest.json
  logs/
    01-foundation.log
    02-frame.log
    03-cabin.log
  steps/
    01-foundation.glb
    02-frame.glb
    03-cabin.glb
    01-foundation.png       # optional
```

GLB and render files are written to a per-step staging directory, hashed after
the Blender process exits successfully, and moved into their final names with
no-overwrite semantics. Any pre-existing run directory, including one with
only `events.ndjson` or abandoned temporary files, is rejected; reruns use a
new `RunId`. Staging is cleaned on failure, while already-created immutable
final steps are retained for forensic inspection and are never described by a
manifest. `manifest.json` is written to a sibling temporary file, flushed,
then atomically renamed into place. The manifest rename is the irreversible
commit point: before it, any error emits `Fail` and no manifest is present;
after it, hash and path verification have already passed and the manifest's
status is `complete`. The `Complete` event is appended after the rename; a
crash or event-log failure at that point leaves the manifest authoritative and
is recovered as complete on the next inspection, without deleting or
rewriting it. A manifest is never published if any required step fails before
the commit point. `verify` recomputes all recorded hashes, checks retained
process logs are present and non-empty, checks paths stay inside the run
directory, checks the source and verified object ID sets recorded by the
bridge, checks timestamps, and rejects duplicate step paths. Reimport is
performed by the real stage
integration command before the manifest is written; `verify` does not need
Blender and validates the recorded proof plus the design's expected ID set.

## Lifecycle and Failure Behavior

The transition rules are strict: `Start` is accepted only from `Idle`,
`Checkpoint` and `Heartbeat` only from `Running`, `Complete` only after all
ordered steps are checkpointed, and `Fail` from any nonterminal state.
Terminal states reject duplicate completion or failure. The pipeline emits
`Start`, then a heartbeat immediately and every 5 seconds while each Blender
process is alive, one checkpoint after each immutable artifact is verified, and
`Complete` only after the atomic manifest rename. Any bridge, hash, validation,
or filesystem failure before the commit point emits `Fail` with the current
step and leaves no final manifest. The server maps the same events into SSE
snapshots. A run is stale
when its status is `running` and `now - lastHeartbeatAt > 15 seconds`; a
one-second server monitor recalculates that derived flag and emits a snapshot
even when no lifecycle event arrives. Staleness does not silently complete or
fail the run. The existing replay UI remains available independently of
artifact execution.

The bridge protocol is fixed. The F# process invokes:

```text
<python> stage/blender/fixture.py --step-id <id> --output <tmp.glb>
  --result-output <tmp.result.json> [--render-output <tmp.png>]
```

`<python>` comes from `--python` or `SOMHAIRLES_BPYTHON`; the default is
`python3`. The script must import `bpy` and report version `>= 3.0.0`. Exit 0
writes exactly one JSON result object to `--result-output` containing `ok`,
`blenderVersion`, `exportedAt`, `objectIds`, `verifiedObjectIds`, `glbBytes`,
and optional `renderPath`/`renderedAt`. Standard output and standard error are
diagnostic streams only; the bridge retains both in the step log so incidental
Blender output cannot corrupt result parsing. Exit 2 means invalid step or
arguments, exit 3 means missing/unsupported `bpy`, and exit 4 means export or
render failure.
The bridge timeout is 120 seconds per step. `exportedAt` is emitted by Python
immediately after the successful GLB export; F# rejects a missing or non-UTC
timestamp. `verifiedObjectIds` comes from reimporting the just-written GLB and
must equal the design's expected IDs.

## CLI and Server Surface

The CLI exposes:

```text
dotnet run --project src/SomhairlesDream.Cli -- run [options]
dotnet run --project src/SomhairlesDream.Cli -- verify <manifest.json>
```

`run` accepts `--project-id`, `--run-id`, `--base-id`, `--target-id`,
`--output-root`, `--stage-root`, `--python`, and `--render`. Defaults are the
IDs above, `.artifacts`, `stage`, `SOMHAIRLES_BPYTHON`/`python3`, and no
render. It writes `events.ndjson` and prints the final manifest path. Exit 0
means complete, 2 means invalid arguments, 3 means bridge failure, 4 means
artifact/manifest failure. `verify` exits 0 only when every hash and contract
check passes, and exits 4 otherwise.

`RunSnapshot` is also defined in `SomhairlesDream.Shared` and is the single
JSON/SSE projection used by the server and browser. The server accepts JSON
bodies with the four IDs and optional `render` on
`POST /api/runs/start`; it returns HTTP 202 with the accepted `RunSnapshot` in
the response body, or 409 for an already-running `(ProjectId, RunId)`. Invalid
JSON or IDs returns 400;
unknown manifest/run selectors return 404; a terminal run returns 409 for a
second start. It exposes
`GET /api/runs/events?projectId=...&runId=...` for that exact run and
`GET /api/artifacts/manifest?projectId=...&runId=...` for its completed
manifest; that endpoint returns 200 with the manifest after completion and 409
while the selected run is still active. The start endpoint launches the same
executor asynchronously. Checkpoint, heartbeat, complete, and fail are
internal pipeline/store calls, not public HTTP commands, so callers cannot
bypass artifact validation to force a terminal state. Only one run with a
given project/run pair may be active; different run IDs may execute
independently.

The SSE `RunSnapshot` schema is explicit: `projectId`, `runId`, `baseId`,
`targetId`, `status` (`idle|running|complete|failed`), optional
`currentStepId`, optional `stepIndex`, `totalSteps`, optional
`lastHeartbeatAt`, `isStale`, optional `error`, and optional `manifestUrl`.
The query parameters prevent event multiplexing between runs.

On successful completion the browser's telemetry panel displays the current
step and a link with ID `artifact-manifest-link` to the manifest endpoint. A
failed or stale run displays the error state and no link.

Code publication is separate from runtime artifact publication. This task has
explicit authorization to publish after all verification passes. The scoped
commit set is the project code, tests, stage fixture, and design documentation
on `main`; no unrelated worktree files are staged. Create only
`/home/somhairle/git/somhairles-dream-fsharp.git` with `git init --bare`, add
local-path origin `/home/somhairle/git/somhairles-dream-fsharp.git`, and push
`main` without force. Verify `git ls-remote` contains the pushed ref and
verify the public cgit tree at
`https://git.somhairle.bid/somhairles-dream-fsharp/` (or the URL discovered
from the cgit response). If that path already exists, do not reinitialize or
overwrite it: inspect that it is a bare repository with the expected project
ref, otherwise stop and report the exact boundary. No Docker, nginx, cgit
configuration, auth files, or other repositories are changed.

## Verification

- Shared tests cover IDs, lifecycle transitions, manifest shape, and path/hash
  validation.
- Modeling tests use a fake bridge for deterministic atomic-writer tests.
- A stage integration command runs the real `bpy 5.0.1` fixture, verifies all
  three GLBs and optional renders, then runs CLI `verify`.
- Server verification checks `/health`, lifecycle JSON/SSE, and manifest
  serving.
- Browser verification checks SSE reaches completion, exposes the manifest
  link, and preserves the already-fixed `Uint32BufferAttribute` mesh path.

## Alternatives Considered

1. **CLI-only pipeline:** smallest server change, but HTTP and CLI behavior
   could diverge and the browser could not observe real run progress.
2. **Server-owned Blender process:** direct integration, but it couples the
   web host to stage tooling and makes CLI verification harder.
3. **Shared library plus thin CLI and server adapter (chosen):** one typed
   pipeline implementation, isolated Blender fixture, and both local and HTTP
   execution paths with explicit boundaries.