[source]

[source] says where the rows come from and how each row becomes one chunk: its stored text, its optional image, its key and its position. This page also covers [embedding.image_input], which chooses what image-space models embed.

[source]Copy Anchor Link

Paths resolve against the build's current directory and expand ${VAR} and a leading ~/ (see the spec file). A key that does not apply to the declared kind is ignored without a message.

Validation errors:

KindsCopy Anchor Link

sqliteCopy Anchor Link

Runs query on the database opened as file:<db>?mode=ro. Each result row becomes a map of its columns. Joins run next, then derive, then sorting and templates.

csvCopy Anchor Link

Reads path with a header row. Every value is a string.

jsonlCopy Anchor Link

Reads path as one JSON object per non-blank line.

image_dirCopy Anchor Link

Finds every .jpg, .jpeg, .png, .bmp, .webp, .tiff and .tif file under input_dir (extensions match in any case), sorted by path. Each image is one row: the key is its path relative to input_dir, the stored text is that path, or <path> [<label>] when it has a label, and the source_uri is item://<name>/<path>. order_by, text, image, joins and derive are ignored.

pdf_dirCopy Anchor Link

Accepted by validation. Loading the rows always fails.

spec error: source.kind=pdf_dir: build via forge_pipeline (pages are temporary)

Labels for image_dirCopy Anchor Link

labels is either a .csv file or a JSON object:

• A .csv file: the first row is a header and is skipped; each following row maps its first column (image id) to its second (label).
• Any other extension is read as a JSON object mapping image id to label.

An image takes the label keyed by its relative path, else the one keyed by its file name without extension. A label file that does not exist fails with a Python FileNotFoundError (exit 1).

[source.text]Copy Anchor Link

Rendering rules:

• A column the row does not have renders as an empty string.
• The template is trimmed and split into lines. A line whose placeholders all render empty is dropped, so optional fields leave no blank lines.
• On each kept line: empty () and empty [ ] or [,] are removed, a trailing , ] becomes ], an em dash (U+2014) at the start or end of the line is removed, and runs of whitespace collapse to one space.
• Lines are joined with newlines.

An empty template, or a template that renders to nothing for a row, falls back to identity-only text: the row's label when [source.image] label_template gives one, else its key. The manifest records text_quality = "template" when a template is set and "identity-only" when it is not.

[source.text]
template = """
{name} ({category})
{description}
"""

[source.image]Copy Anchor Link

Attaches one image file to each row of a sqlite, csv or jsonl source.

When path_template is set, every row's rendered path must be an existing file, or loading stops with spec error: source.image: file missing for key (...): <path> (exit 2). The file's SHA-256 is taken at load and used for deduplication.

The label enters the row's input hash, the manifest items, the identity-only text fallback, and the utility queries of the crf gate.

Setting path_template is what allows a sqlite, csv or jsonl source to use image = "space" models. Without it, validation fails with models.<preset>.image=space: source declares no images.

[[source.joins]]Copy Anchor Link

sqlite only. Each entry adds columns from a second query to the main rows.

Both keys have no default: an entry without one fails with a Python TypeError (exit 1), not a spec error.

[[source.joins]]
query = "SELECT sku, category, brand FROM product_meta"
on = "sku"

How a join applies:

1. The join query's rows are indexed by their on value. When two join rows share a value, the later one wins.
2. Each main row whose on value is in the index gains the join row's columns. A column the main row already has keeps the main row's value.
3. Joins run in order, before [source.derive].

Join errors (exit 2):

• spec error: source.joins: join query has no column '<on>'
• spec error: source.joins on '<on>': 0 of N rows matched - check the key name and its sqlite type on both sides, when the join returned rows and none matched. A TEXT key never matches an INTEGER key; cast one side.

A partial match is not an error. It prints [forge] join on '<on>': H/N rows matched on stdout.

[source.derive]Copy Anchor Link

sqlite, csv and jsonl. Each entry adds a computed column:

[source.derive]
image_stem = "basename_stem(image_url)"
name_lower = "lower(name)"

Derived columns are computed per row after joins and before sorting, so they work in order_by, templates and path_template. The expression must be exactly helper(column). Errors fire when rows load (exit 2): spec error: source.derive: unsupported expression '<expr>' (use helper(column)) and spec error: source.derive: unknown helper '<h>' (valid: basename_stem, lower).

Row identityCopy Anchor Link

These rules fix which chunk each row becomes and what its citation depends on.

• Key: the order_by values as strings, joined with a vertical bar. For image_dir, the relative path.
• Order: rows sort by the tuple of their order_by values as strings. Numeric ids sort as text, so 10 comes before 2. Zero-pad them to sort numerically.
• Uniqueness: two rows with the same key stop the load with spec error: source.order_by [...] is not a total order (duplicate key (...)); append a unique column (RFC-0 N1).
• Ordinal: the row's position after sorting (and after --sample, which renumbers). The chunk's span is [ordinal, ordinal + 1).
• source_uri: a non-empty column named source_uri in the row, else item://<corpus.name>/<key>.
• Item input hash: SHA-256 over the stored text's digest, the image's SHA-256 (or none), the label and chunker_version.
• corpus_input_hash: SHA-256 over the item input hashes in row order.

The span enters the chunk_id, so inserting or deleting a row changes the chunk_id of every row sorted after it. A --sample build renumbers ordinals and never shares citations with the full build. See citations and hashes.

Deduplication: rows whose source images are byte-identical share one frame and one image vector. This happens when [media] dedup = true (the default) or when the spec has no [media]. Text vectors are always computed per row.

[embedding.image_input]Copy Anchor Link

Chooses what image = "space" models embed.

• source embeds the original image files.
• decoded_media embeds the frames decoded from the encoded media, so the vectors match what the corpus stores. It adds the decoder settings (backend, canvas, crf, pixel format) to those models' recipes.

Errors: embedding.image_input.mode: source | decoded_media for any other value, and embedding.image_input.mode=decoded_media requires [media].

The resolved mode enters every model's embedding recipe, text-only models included. Changing it, or adding or removing [media] while mode is "", recomputes every model's vectors once. Other keys under [embedding] are ignored without a message.