Build presets

A build preset picks four storage settings at once: the text encoding, the embedding dtype, whether an HNSW index is written, and whether a BM25 index is written. Presets are resolved by the Python bridge behind urna.build; the Rust writer has no preset concept.

PresetsCopy Anchor Link

index_type and rerank are the values urna stats prints (the manifest's index_type and rerank_policy). The rerank source is what urna ask --disclose explain reports as real cosine (full precision) or real cosine at stored precision, and what urna retrieve writes as rerank_source. Only float32 counts as full precision: no 0.5.1 writer emits the full-precision slab (section 0x09), so float16, int8 and int4 files rerank at the precision they store. See Presets and stored precision.

Preset names are case-sensitive. Any other value fails:

ValueError: unknown preset: micro (expected exact|compressed|tiny|nano|hybrid)

What each preset writesCopy Anchor Link

Every file has the six required sections (0x01 chunk ids, 0x02 canonical text, 0x03 spans, 0x04 embeddings, 0x05 provenance, 0x06 search contract). The preset decides how they are encoded and which index sections follow.

exactCopy Anchor Link

Text sections raw, embeddings float32 raw (4 bytes per component). No index. Every search is a flat scan with recall = 1.0.

compressedCopy Anchor Link

The zstd text path: chunk ids as intpack, canonical text through the text-codec chooser (the smallest of plain zstd, per-chunk zstd streams, zstd with a trained dictionary in 0x0A, fsst, or deduplication with a back-reference map in 0x0B), spans as the smaller of intpack and zstd, provenance and search contract as zstd. Embeddings float16 (2 bytes per component). No index.

tinyCopy Anchor Link

The zstd text path, embeddings int8 (1 byte per component plus one float32 scale per vector), and an HNSW index in 0x07. Searches through ask and retrieve take the HNSW route, then rerank the shortlist exactly.

nanoCopy Anchor Link

The zstd text path, embeddings int4 in blocks of 64 components (half a byte per component plus one float16 scale per block), and an HNSW index. The effective dim must be a multiple of 64:

ValueError: int4 requires (effective) embedding_dim divisible by 64, got 96

hybridCopy Anchor Link

The zstd text path, embeddings float32, an HNSW index in 0x07 and a BM25 index over the canonical text in 0x08 (zstd). The manifest sets supports_bm25 = true and, as the callout above says, index_type = "hnsw". The quickstart corpus is built with this preset:

index_type:   hnsw
rerank:       exact
supports_ann: true
supports_bm25:true
sections:     9
  0x01 chunk_ids                encoding=intpack  389 bytes
  0x02 chunks_canonical         encoding=zstd     1484 bytes
  0x03 chunks_original_spans    encoding=zstd     154 bytes
  0x04 embeddings               encoding=raw      12288 bytes
  0x05 provenance               encoding=zstd     118 bytes
  0x06 search_contract          encoding=zstd     95 bytes
  0x07 hnsw_index               encoding=raw      182 bytes
  0x08 bm25_index               encoding=zstd     1695 bytes
  0x0c graph_adjacency          encoding=raw      174 bytes

The 0x0C graph comes from the forge's [build] with_graph = true, not from the preset.

HNSW parametersCopy Anchor Link

The indexed presets build HNSW from the float32 vectors (after any mrl_dim truncation, before quantization) with hnsw_m = 16, hnsw_ef_construction = 400 and hnsw_seed = 42 unless you pass other values to urna.build. The seed makes the index bytes deterministic.

micro is a recipe, not a presetCopy Anchor Link

micro is the published name of one point on the size curve: int8 embeddings truncated to their first 256 components. preset="micro" raises ValueError. Build it with explicit settings:

urna.build(
    "out/corpus-micro.urna",
    embedding_model, embedding_dim, chunker_version, model_hash, chunks,
    text_encoding="zstd", dtype="int8", mrl_dim=256, with_hnsw=True,
)

The file records mrl_dim = 256 and full_dim (the source dim), and urna stats prints both. micro is int8, so the multiple-of-64 rule does not apply to it; it applies to any int4 build, truncated or not.

OverridesCopy Anchor Link

The preset only sets defaults. Explicit arguments win:

with_hnsw=True on any preset sets index_type = "hnsw" and rerank_policy = "exact".

Where presets are chosenCopy Anchor Link

The build spec does not validate [build] preset or [build] dtype itself: a bad value fails inside urna.build at the emit stage, after the embeddings are computed. See [build] and [output].

In Rust, call the builder methods directly. The hybrid preset corresponds to:

UrnaFileBuilder::new(manifest)
    .text_encoding(SectionEncoding::Zstd)
    .embedding_dtype(EmbeddingDType::Float32)
    .hnsw_index(HnswIndex::build(vectors, n, dim, 16, 400, 42).to_bytes())
    .bm25_index(Bm25Index::build(&docs, DEFAULT_K1, DEFAULT_B).to_bytes())

Adding .hybrid() declares index_type = "hybrid" and score_type = "hybrid_rrf"; only then do ask, retrieve and search-text take the hybrid route.

Presets and citationsCopy Anchor Link

The embedding dtype and mrl_dim are part of content_hash, and so is the search contract. Two builds of the same rows with different presets have the same chunk_id values but different content_hash values, so a urna://content_hash/chunk_id citation from one file does not resolve against the other. The text encoding alone does not move content_hash: every text codec decodes to the same bytes. See Citations and hashes.

Measured sizesCopy Anchor Link

The repository publishes one measured run of every preset in data/measure/ladder.json.

Conditions:

• Corpus: data/corpus_next.v1.urna, 30,725 chunks at dim 384, built with a MiniLM model that is not trained for matryoshka truncation.
• 100 queries, k = 10, seed 0, NEON SIMD backend, python/tools/measure_presets.py.
• recall@10 is measured against the float32 exact top 10 on a self-perturbation ruler: each query is a corpus chunk's own embedding plus tiny noise. It measures how stable the ranking stays under quantization, not retrieval quality on real queries, and it is likely inflated.
• The hybrid row was measured by calling search_hybrid directly. urna ask on a hybrid file takes the HNSW route instead.

More points of the truncation curve are on Presets and stored precision.