Build artifacts

This page describes every file urna build writes, the embed cache it shares across builds, the reproduction levels the build lock supports, and the catalog of build errors.

FilesCopy Anchor Link

A build writes into [output] dir (or --out-dir). For a corpus named handbook with media on the av1 backend:

Each .urna is written into .tmp/, moved into place with an atomic rename (same filesystem), reopened and validated before the build continues. The per-model files have no manifest or media directory of their own: they share <name>.manifest.json and <name>.media/, so a tool that derives those names from a per-model file's name finds nothing.

The .urna fileCopy Anchor Link

Each chunk is one row: its stored text, its source_uri, and the span [ordinal, ordinal + 1). Space 0 holds the text = "default" model's vectors at the [build] preset. The file's provenance records {dataset: <name>, corpus_input_hash}, and its manifest takes title and version from [corpus].

Depending on the spec the file also has the chunk graph (with_graph, on by default), the space table and one band per named space, and, with media, the blob references and a span overlay that points each chunk at its frame or file. With embed_media = true the media bytes are stored in the file too. The graph, the space table and bands, and the media tables are outside the content_hash: two files with the same chunks and the same space 0 have the same content_hash, whatever named spaces or media tables they add. The sections are described on layout.

<name>.manifest.jsonCopy Anchor Link

The build record, manifest_schema_version 1. It is written atomically as JSON with sorted keys and one-space indentation, and the writer refuses to save it without its required fields. It is separate from the manifest stored inside the .urna.

An excerpt of the quickstart manifest:

{
 "corpus_input_hash": "sha256:00ebc2f0e96cbbf0effd0ce5b38a80f8b8eb13c972fdcde4968affa21caaa8bb",
 "items": [
  {
   "image_path": null,
   "key": "01-what-is-urna#1",
   "label": null,
   "media_uri": null,
   "ordinal": 0
  }
 ],
 "items_compact": false,
 "manifest_schema_version": 1,
 "media": null,
 "models": {
  "potion": {
   "embedding_recipe_hash": "sha256:f5c4ef99c235a02f1185b0830a16af090c5e3fdb28e6b20f2863786841539d45",
   "items_per_s": 6270.29,
   "model_hash": "sha256:8f2eb91a754b4da59cdd8223d0ba196185fed1bb6f092fd2be0ff02b893b1c98",
   "recipe": {
    "adapter_version": 1,
    "device_class": "auto",
    "encode_kwargs": {},
    "image_doc_format": "dict",
    "image_input_mode": "source",
    "image_max_side": 0,
    "image_mode": "image_document",
    "image_prompt": "Represent this image.",
    "model_dtype": "",
    "normalize": true,
    "preprocess_version": "processor-native-v1",
    "preset": "potion",
    "text_corpus_mode": "plain",
    "text_query_mode": "plain"
   }
  }
 },
 "n_items": 12,
 "n_unique_frames": 12,
 "name": "quickstart",
 "outputs": {
  "quickstart.urna": {
   "build_s": 0.056,
   "bytes": 17942,
   "file": "out/quickstart.urna",
   "file_hash": "sha256:e4d5f8907faad38c192dc6929e36dbf16db558410b2dae5abb4d65f10f508832"
  }
 },
 "provenance_mode": "standard",
 "spaces": [],
 "sql": "",
 "text_quality": "template",
 "timings": {
  "embed.potion": 0.002,
  "rows": 0.001
 }
}

The quickstart has twelve items; the excerpt keeps the first. The file path is relative to the spec's directory because provenance is standard.

The media recordCopy Anchor Link

The record depends on the backend. Every backend records source_bytes as the size of the original files, so compression_ratio compares across backends.

The build adds dedup (n_items, n_unique_frames), order and order_permutation when av1 applied an ordering, crf_auto (the crf gate report) when crf = "auto", and embedded (the embed_media value).

Reading itemsCopy Anchor Link

Under minimal, items holds only key and ordinal. The forge's reader forge_manifest.manifest_items(manifest, need=(...)) raises an error naming provenance = "standard" when a caller asks for a dropped field, and forge_manifest.frame_resolver(manifest) maps an item to its frame in the media stream in every mode. Both live in the checkout.

<name>.build.lock.jsonCopy Anchor Link

The environment record behind a byte-identical rebuild, lock_schema_version 1.

The tool version comes from running the tool with -version. Only ffmpeg and ffprobe accept that flag; for cjxl, djxl and ssimulacra2 the field holds their usage or error text, such as Unknown argument: -version. The sha256 still identifies the binary.

The lock records every tool and package on every build, text-only builds included.

What --rebuild-only comparesCopy Anchor Link

--rebuild-only builds a new lock and compares it with the one on disk, key by key. It ignores exactly these keys, because they only say where things live: resolved_spec.spec_path, resolved_spec.output.dir, resolved_spec.output.cache_dir, resolved_spec.source.path, resolved_spec.source.db and resolved_spec.source.image.path_template. Everything else counts: a new OS release, a changed tool binary or package version, a different URNA_ST_DEVICE. source.input_dir and source.labels are not ignored, so an image_dir corpus rebuilt under a moved data root diverges on location alone.

Each difference prints [forge] warning: build.lock divergence (L3 not claimable): ... (up to eight listed). With the Python tool's --strict-env it is an error instead (exit 1).

Shared embed cacheCopy Anchor Link

Vectors are cached per model under the cache root (by default ~/.cache/urna), shared by every spec and output directory:

~/.cache/urna/
  embed/potion/76370c3d6d796e47.npz
  embed/potion/76370c3d6d796e47.npz.sha256
  embed/potion/76370c3d6d796e47.npz.lock
  models/model_hash.potion.efe05bc4cb560530.json

• embed/<preset>/<16 hex>.npz holds one model's vectors for one set of rows: text (one vector per row) and/or image_unique (one per unique frame), plus meta with the three hashes below. The file name is the first 16 hex digits of a hash over model_hash, embedding_recipe_hash, corpus_input_hash and which arrays the spec needs (text, image, dedup). A changed knob adds a new entry beside the old one.
• A cache entry is used only when the three hashes in meta match and the .sha256 sidecar matches the file. Writes take an exclusive lock on the .lock file and land through a temporary file and a rename; a torn or mismatched entry is recomputed, never reused.
• models/model_hash.<preset>.<16 hex>.json caches a model's model_hash so a cache hit does not need to load the model. It is keyed by the preset plus normalize, dtype, device and model_path, and holds model_hash, a fingerprint of the model directory's file listing, and those knobs. It is trusted while the directory listing matches. potion, open_clip presets and fake-test have no model directory to fingerprint, so replacing their weights does not invalidate the cached hash until the model is loaded again. When a loaded model's hash disagrees, the build rewrites the file.

The recipe hashed into embedding_recipe_hash holds adapter_version, preset, image_input_mode, text_corpus_mode, text_query_mode, image_mode, image_prompt, normalize, preprocess_version, image_max_side, image_doc_format, encode_kwargs, model_dtype and device_class. For image-space models embedding decoded media it also holds the decoder settings (backend, canvas, crf, pixel format, toolchain hash), so a text-only model's entry survives a change of media crf.

Deleting the cache is safe: the next build recomputes it.

Reproduction levelsCopy Anchor Link

The build declares three levels of reproduction:

The build supports L3 with reproducible = true (the default, which pins the file's creation time), fixed encoder parallelism, and --rebuild-only, which re-emits from the cached vectors and compares the lock. The byte comparison is left to you: record file_hash from the first build's result or manifest and compare it after the rebuild. See reproducible builds.

Error catalogCopy Anchor Link

How a failure surfaces depends on when it happens:

At parse timeCopy Anchor Link

All exit 2, before anything else runs:

• unsupported spec extension '<ext>' (use .toml or .json)
• yaml specs need pyyaml: pip install pyyaml (or use .toml)
• <key>: ${NAME} is not set; export NAME=/path (...), or is empty
• unknown top-level table(s) [...]: valid tables are [...]
• models must be an array of tables; write [[models]], not [models]
• <table>: unknown key '<k>' (valid: ...)
• media.profile: unknown '<p>' (valid: [...])

At validationCopy Anchor Link

Exit 2, also under --dry-run: every rule on the [corpus], [source], [[models]], [media] and [output] pages. An unknown preset, or fake-test without URNA_ENABLE_FAKE_PRESET=1, is a registry error (exit 4) even here.

While the build runsCopy Anchor Link

Spec errors raised after work may already be done (exit 2):

Registry errors (exit 4):

• unknown model preset '<p>'. valid presets: ...
• preset 'fake-test' requires URNA_ENABLE_FAKE_PRESET=1 (test-only)
• preset '<p>' needs the '<module>' package. install with: <pip line>
• A pinned model-repository file that is missing (pinned code file missing: <file>) or altered (... does not match the pinned allowlist (...). refusing to execute unreviewed remote code.)
• A model worker that fails or dies, including a bad [[models]] dtype

Build errors, each ending a traceback (exit 1):

• source produced no rows
• media enabled but N rows have no image (first: <key>)
• --rebuild-only: cache for '<p>' is missing or stale (triad mismatch); run a full build
• build.lock divergence (L3 not claimable): ..., only with --strict-env on the Python tool
• media gate/cluster model '<p>' is not a spec model

Other failures with a traceback (exit 1): missing media tools (media.crf="auto" needs ssimulacra2 on PATH: brew install jpeg-xl, jxl backend needs 'cjxl' on PATH), encoder failures (encoder wrote yuv420p for a requested yuv444p ..., encoded N frames for M images ...), SQLite errors, a missing CSV or JSONL file, values that urna.build rejects ([build] preset, dtype, mrl_dim), value types the spec does not check, the crf = "auto" crash on rows without a label, and a sentence-transformers preset with no weights on disk.