Sections

A section is one payload named by a u32 id in the section table. This page lists every id that format v1 defines, what writes it, what reads it, and the byte layout of each payload.

Required and optionalCopy Anchor Link

Six sections are required. They are also the canonical sections: content_hash covers exactly these six, in this order, and a reader rejects a file missing any of them with MissingRequiredSection.

The order is alphabetical by name and fixed by spec, so a new section id can never reshuffle the hash. Every other section is optional and outside content_hash. Adding one of them changes file_hash but not the citations, with one exception on Hashes and citations: attaching HNSW also rewrites search_contract.

Section id mapCopy Anchor Link

"Text encodings" means the set {0 raw, 1 zstd, 4 intpack, 5 zstd_dict, 9 fsst, 10 txt_streams}. "Vector encodings" means {0 raw float32, 2 float16, 3 int8, 7 int4}. The reader applies the vector set to 0x04 and to both band ranges, and the text set to every other id. Encoding ids are on Encodings.

Names in the table for 0x09 to 0x13 come from the Rust constants (SECTION_EMBEDDINGS_FP, SECTION_DICTIONARY, SECTION_CHUNK_SCALARS and so on); they are not stored in the file. section_name() does not resolve those ids, so urna inspect --json labels them "unknown". 0x0A and 0x0B stay unnamed on purpose: they are private companions of 0x02.

"Written by" names the UrnaFileBuilder method. The HNSW and BM25 index sections only open on section presence; the manifest flags supports_ann and supports_bm25 are not consulted by the runtime. A 0x0C section without graph_present is ignored, and search_graph then falls back to exact search.

Who writes whatCopy Anchor Link

The builder takes optional payloads as opaque bytes. It does not decode or validate them, so a bad HNSW, BM25, graph, blob or space payload builds and fails later, when MmapUrnaFile::open decodes it. See urna.build and the spec file for the Python and forge options.

Space bandsCopy Anchor Link

• Band 0x20 + i holds the vectors of space i. Space 0 is the text space in 0x04 and is never listed in 0x15, so usable band ids are 0x21 to 0x2F: at most 15 named spaces.
• A space_table entry with space_index 0 or 16 and above is rejected. The builder's space_band(index, ...) does not range-check its index.
• Every band listed in 0x15 must be present, with the exact size its (n_vectors, dim, dtype) implies, and n_vectors must equal n_chunks: bands are parallel to the chunks.

Payload formatsCopy Anchor Link

Most sections open with a common 12-byte prefix: u32 version = 1 followed by u64 count. Strings are length-prefixed: u32 len then UTF-8 bytes, no NUL. Decoders reject trailing bytes and bound every count against the remaining bytes before allocating.

0x01 chunk_idsCopy Anchor Link

The writer uses intpack under zstd text and falls back to raw when any id is not lowercase sha256:<64 hex>.

0x02 chunks_canonicalCopy Anchor Link

Raw: prefix + count strings, the canonical text of each chunk. Under zstd text the writer picks the smallest of five codecs; all of them decode to these exact raw bytes. See the text-codec chooser.

0x03 chunks_original_spansCopy Anchor Link

Under zstd text the writer keeps the smaller of intpack and zstd.

0x04 embeddingsCopy Anchor Link

No prefix. A fixed-stride slab of n_embeddings rows at embedding_dim, in the layout of the manifest dtype: see embedding dtypes.

0x05 provenanceCopy Anchor Link

u32 1 + u64 json_len (the count slot carries the JSON byte length) + compact JSON of any value. The reader only requires it to parse. It is canonical, so its bytes, key order included, move content_hash.

0x06 search_contractCopy Anchor Link

u32 1 + u64 json_len + JSON {metric, score_type, normalize, index_type, rerank_policy}. The writer fills it from the manifest, and the reader rejects a file where the two disagree, with the matching Unsupported* error and the text section says X but manifest says Y.

0x07 hnsw_indexCopy Anchor Link

Payload version 2: seven u32 (payload version 2, m, m_max0, ef_construction, entry_point, max_level, n_nodes), then three columns of u32 len + intpack: the level of each node, the neighbour count per layer, and the neighbour ids in build order. Version 1 payloads still decode.

0x08 bm25_indexCopy Anchor Link

Payload version 2: u32 2, f32 k1, f32 b, f32 avgdl, u32 n_docs, u32 n_terms, intpack of document lengths, then per term (sorted by bytes) a string token and u32 df, then intpack of delta-gapped document ids and intpack of term frequencies. Version 1 payloads still decode.

0x09 embeddings_fpCopy Anchor Link

A fixed-stride slab of n x dim float32 or float16 values. The runtime infers the dtype from size / (n * dim) (4 or 2 bytes). No writer emits this section in 0.5.1.

0x0A dictionaryCopy Anchor Link

The raw trained zstd dictionary that the zstd_dict codec of 0x02 decodes against.

0x0B dedup_mapCopy Anchor Link

u8 0 + intpack of one u32 per chunk: the index of its text in the unique pool stored in 0x02.

0x0C graph_adjacencyCopy Anchor Link

u32 1 + u64 n_nodes + (u32 len + intpack of n_nodes + 1 offsets) + (u32 len + intpack of destination gaps, delta-coded within each (source, edge type) run) + an edge-type column: u8 0 + u8 type when every edge has one type, or u8 1 + u32 len + intpack of types.

Edge types: EDGE_TYPE_NEXT_CHUNK = 0, EDGE_TYPE_SEMANTIC = 1, EDGE_TYPE_CITATION = 2. Edges are sorted by (source, type, destination), and a node has at most GRAPH_MAX_DEGREE = 2^20 edges. The runtime requires n_nodes == n_embeddings.

0x14 blob_refsCopy Anchor Link

u32 1 + u64 n + n x (32-byte raw SHA-256 of the original bytes, string original_uri, u64 byte_len, u8 inlined 0 or 1). Entry order is the contract: the overlay and blob_data address records by position.

0x15 space_tableCopy Anchor Link

u32 1 + u64 n + n x (u8 space_index, string name, u32 dim, u8 dtype, string model_hash, u64 n_vectors). The dtype codes are 0 float32, 1 float16, 2 int8, 3 int4. Duplicate indices or names are rejected, and model_hash must start with sha256:.

0x16 blob_span_overlayCopy Anchor Link

u32 1 + u64 n + n x (u32 blob_ref_index, u64 byte_start, u64 byte_end), one entry per chunk in chunk order. BLOB_REF_NONE = 0xFFFFFFFF keeps the chunk's 0x03 span. At open the runtime replaces the span of every other chunk with the blob-relative range, so hits and cite report offsets inside the blob. A blob_ref_index past the end of 0x14 fails the open.

0x17 blob_dataCopy Anchor Link

u32 1 + u64 n + n x (u64 offset, u64 len), offsets relative to the first data byte, then the concatenated blob bytes. A record kept outside the file has (0, 0). Media is already compressed by its own codec, so this section is always raw.

0x20 to 0x2F bandsCopy Anchor Link

A fixed-stride slab of n_vectors rows at the space's dim, in the layout of its dtype, the same layouts as 0x04.

Accepted, never writtenCopy Anchor Link

The reader has no allow-list of ids. Any id, the reserved names 0x0D to 0x13 and the 0x30 to 0x3F band included, loads when its encoding is legal for its class and its checksum matches. The runtime ignores what it does not know. A future optional section therefore opens in a 0.5.1 reader, as long as it uses an encoding that reader knows.

Encodings and dtype layouts are next: Encodings.