Embedders

urna.build and the search methods take vectors, not text. An embedder turns text into those vectors and reports the identity (embedding_model, model_hash) that the file records. The wheel ships one embedder, urna.embed_potion. A repo checkout adds a stdlib-only lexical embedder and the adapters of the model registry.

urna.embed_potionCopy Anchor Link

The potion-base-8M static table from model2vec, bundled in the wheel. It needs numpy and tokenizers (the embed extra), no torch, no network. It never opens a socket.

pip install "urna[embed]"

from urna.embed_potion import potion_embedder

emb = potion_embedder()
vectors = emb.embed_texts(["first text", "second text"])   # two lists of 256 floats

In a checkout the same module is forge.embed_potion, with the same contents.

Importing the module sets HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE and HF_DATASETS_OFFLINE to 1 and TOKENIZERS_PARALLELISM to false, each only when it is not already set.

Module namesCopy Anchor Link

PotionEmbedderCopy Anchor Link

PotionEmbedder(model_dir: Path | str = MODEL_DIR, normalize: bool = True)

Loading is lazy: the table loads on the first use of embedding_dim, fingerprint(), model_hash() or embed_texts(), and is cached per directory.

model_hash is a method, not a property: call emb.model_hash().

embed_texts tokenizes without special tokens, averages the token rows, L2-normalizes and casts through float32. An empty text maps to the [UNK] row, so it never returns a zero vector.

With the table in the 0.5.1 wheel, model_hash() is:

sha256:8f2eb91a754b4da59cdd8223d0ba196185fed1bb6f092fd2be0ff02b893b1c98

The hash covers the bytes of the table files, so a different table gives a different hash, and corpora built with the old one fail the model gate against the new one.

FailuresCopy Anchor Link

Make a Python-built file answerable by the CLICopy Anchor Link

The installed urna ask and urna retrieve embed the query themselves. They choose the potion embedder when the manifest embedding_model starts with minishlab/potion, then check the name, the dimension and the model_hash. Pass the embedder's identity to urna.build exactly:

urna.build(
    "corpus.urna",
    emb.embedding_model,     # "minishlab/potion-base-8M/v1"
    emb.embedding_dim,       # 256
    "my-chunker/1",
    emb.model_hash(),
    chunks,
)

Then urna ask corpus.urna "your question" works with the installed binary after urna setup. See urna ask.

Other modelsCopy Anchor Link

Any model works with urna.build, as long as you pass its vectors, its name and a model_hash that identifies it. What changes is who can query the file afterwards.

To stamp a file built with your own sentence-transformers model, compute its hash with model_fingerprint (checkout only) and pass the same name as embedding_model and as model_id, because urna search-text fingerprints the model under the manifest name:

import sys

sys.path.insert(0, "python")
from model_fingerprint import compute_model_fingerprint, fingerprint_to_model_hash, resolve_model_dir

name = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
model_hash = fingerprint_to_model_hash(compute_model_fingerprint(resolve_model_dir(name), model_id=name))

The functions are documented on Builder pipeline.

forge.embed_default (checkout only)Copy Anchor Link

The lexical floor: stdlib only, no numpy, no model files. Each token maps to a fixed pseudo-random vector derived from its bytes, and a text is the L2-normalized mean of its token vectors. Cosine therefore reflects shared literal tokens only: synonyms do not land close. Use it for tests and for environments with nothing but the standard library.

import sys

sys.path.insert(0, "python")
from forge.embed_default import StaticEmbedder

emb = StaticEmbedder()
emb.embedding_model   # "urna-forge-static/v1"

The seed and the dimension are part of the fingerprint, so changing either changes model_hash. The default instance hashes to sha256:75acb1edc57eb9dcb18d989fac562bb2575e20d3de211d849c4dbf649b81d1a1.

The tokenizer splits on alphanumerics, which works poorly for CJK text.

The forge packageCopy Anchor Link

import forge (checkout only) exports:

Embedder protocolsCopy Anchor Link

Two shapes of embedder exist, and they are not interchangeable.

forge.retrieve.retrieve(..., embedder=...) calls model_hash() as a method, so it needs the static protocol. builder.Pipeline(embedder=...) accepts any callable that maps a list of chunk specs to a list of vectors. With a registry adapter, pass role="query" when you embed a query.

The registry itself (presets, dimensions, the remote-code and heavy gates) is on Model registry. The protocol the CLI uses to run an embedder script is on Query embedder protocol.