The spec file

A spec file (by convention corpus.toml) declares one corpus build for urna build --spec: where the rows come from, which models embed them, how media is encoded and what files come out. This page covers the file as a whole and the [corpus] table. Each other table has its own page.

FormatsCopy Anchor Link

The format follows the file extension:

Any other extension fails with spec error: unsupported spec extension '<ext>' (use .toml or .json).

The same spec in TOML and in JSON:

[corpus]
name = "quickstart"
chunker_version = "quickstart/1"

[source]
kind = "jsonl"
path = "examples/quickstart/docs.jsonl"
order_by = ["id"]

[source.text]
template = "{text}"

[[models]]
preset = "potion"
text = "default"

{
  "corpus": { "name": "quickstart", "chunker_version": "quickstart/1" },
  "source": {
    "kind": "jsonl",
    "path": "examples/quickstart/docs.jsonl",
    "order_by": ["id"],
    "text": { "template": "{text}" }
  },
  "models": [{ "preset": "potion", "text": "default" }]
}

TablesCopy Anchor Link

A spec has at most seven top-level tables. Any other top-level name fails with spec error: unknown top-level table(s) [...]: valid tables are [...].

models must be an array of tables. Writing [models] instead of [[models]] fails with spec error: models must be an array of tables; write [[models]], not [models].

Environment variablesCopy Anchor Link

Before parsing, every string value in the spec is expanded, including SQL queries and templates:

1. ${NAME} is replaced with the value of the environment variable NAME. Only the braced form expands, so a bare $ in a query and the {column} placeholders of a template are left alone.
2. Then a leading ~/ becomes your home directory. A variable that holds ~/data works, because this step runs after substitution. A ~ anywhere else in the string is left alone.

An unset or empty variable stops the build and names the key where it appears, list positions included:

spec error: source.db: ${DATA} is not set; export DATA=/path (spec paths stay portable, see docs/USAGE.md section 13)
spec error: models[1].model_path: ${WEMM_DIR} is empty; export WEMM_DIR=/path (spec paths stay portable, see docs/USAGE.md section 13)

Write data roots as variables (db = "${DATA}/shop.sqlite") so the same spec builds on any machine. The build lock records the expanded values.

PathsCopy Anchor Link

Relative paths resolve against the current directory of the build, not against the directory of the spec file. This applies to source.path, source.db, source.input_dir, source.labels, the rendered source.image.path_template, output.dir and model_path. Run the build from the directory the paths are written for, or use ${VAR} and absolute paths.

output.cache_dir, --cache-dir and URNA_CACHE_DIR also expand a leading ~.

Unknown keysCopy Anchor Link

A key that is not part of a table fails with a message that lists the valid keys:

spec error: output: unknown key 'dirr' (valid: allow_remote_code, cache_dir, dir, embed_media, mode, provenance)

Three places behave differently:

• [embedding] reads only image_input. Any other key under [embedding] is ignored without a message.
• [corpus] also accepts the keys source, image_input, media, models, build, output and spec_path, and then discards their values.
• [source.derive] is a free map: its keys are the names of the columns you add.

A [source] key that does not apply to the declared kind (for example query on a jsonl source) is ignored without a message.

When a spec is checkedCopy Anchor Link

A spec is checked in three stages. The earlier a problem is caught, the less work is lost.

A dry run (urna build --dry-run) stops after validation. It never opens the source, so a missing file, bad SQL, an order_by key that is not unique and a join that matches nothing all pass a dry run.

Value types and table shapes are not checked. A value of the wrong type is accepted and fails when the build first uses it, as a Python exception with a traceback and exit 1, not as a spec error. Examples: [[source.joins]] without on raises TypeError; text = "..." directly under [source] instead of [source.text] raises ValueError; media.width = "wide", an unknown media.pix_fmt, an unknown bucket in media.quality.buckets, an unknown build.preset or build.dtype, and an out-of-range build.mrl_dim all pass validation and fail later.

[corpus]Copy Anchor Link

Names the corpus and pins the chunker identity.

Because name enters the provenance and the default source_uri, renaming a corpus changes its content_hash and, for rows without a source_uri column, their chunk_ids. See citations and hashes.

The next table to write is [source].