Contributing

This page covers how to set up a checkout of hoffresearch/urna, what the release gate and CI run, how to fuzz a decoder change, and the rules a pull request has to follow.

How a change landsCopy Anchor Link

1. Fork the repository and branch from main, for example git checkout -b feature/short-description origin/main.
2. Keep each pull request to one concern.
3. Add or update tests. New behavior needs a new test, written against real artifacts (built .urna files, the golden fixtures, real corpora) rather than mocks, covering the happy path, the error path and one edge case.
4. Run scripts/release_check.sh locally before you push.
5. Open the pull request against main and fill in the template. Its description becomes the squash commit.

The rules on main: squash merge only, one approving review, signed commits and linear history, and the branch is deleted after the merge. CI results are not a required status check, so run the gate yourself. Sign your commits with an SSH or GPG key registered on GitHub (git config commit.gpgsign true). Write commit messages in plain English, without a conventional-commits prefix; the message explains why, the diff shows what.

Set up a checkoutCopy Anchor Link

You need:

• Rust 1.88 or newer, edition 2024. The CLI crate (urna) sets rust-version = "1.88"; urna-format, urna-runtime and urna-python keep the workspace's 1.85, but cargo build --workspace builds the CLI, so 1.88 is the floor for a checkout.
• A C compiler, because zstd-sys compiles C.
• Python 3.12 or newer.
• Git LFS, for the potion table and the measurement corpus.
• macOS or Linux for the Python side. The gate copies the extension only on those two, and the checkout loader in python/urna.py has no Windows path. The Windows CI job covers the CLI.

git clone https://github.com/hoffresearch/urna.git
cd urna
git lfs pull

python3 -m venv .venv
. .venv/bin/activate
pip install numpy tokenizers pillow ruff

cargo build --release --workspace
PYO3_PYTHON="$PWD/.venv/bin/python" cargo build --release -p urna-python --features pyo3/extension-module
cp target/release/lib_urna.dylib python/_urna.so   # macOS
cp target/release/lib_urna.so python/_urna.so      # Linux

cp scripts/pre-commit .git/hooks/pre-commit && chmod +x .git/hooks/pre-commit

Build the extension with --features pyo3/extension-module. Without it the library links libpython directly, and under a statically linked interpreter such as uv's standalone Python it loads a second runtime and segfaults at import. PYO3_PYTHON pins the build to the interpreter that runs the tests. Rebuild python/_urna.so after any change to urna-format, urna-runtime or urna-python.

numpy, tokenizers and pillow are the forge dependency group in the root pyproject.toml. Registry models and media backends need more; each preset prints its own install line (see the model registry).

If the LFS budget or a missing git-lfs gets in the way, sh scripts/fetch_potion.sh fetches the potion table alone from Hugging Face at a pinned revision and accepts it only when its SHA-256 matches the LFS pointer. Without the real table, embedding fails and urna doctor exits 5. urna setup cannot fix that inside a checkout, because the checkout's python/forge wins the embedder lookup.

The demo datasets under data/demo/ are local only and gitignored; data/demo/Instructions.md has the commands to fetch them. The test suites do not need them.

The pre-commit hook blocks any staged .urna, *.sidecar.jsonl, cohort_*.urna or file under pairs/, except three sanctioned files: data/corpus_next.v1.urna, data/measure/fakerecogna_exact.urna and crates/urna-format/tests/fixtures/golden_v1_minimal.urna. Copy it into .git/hooks rather than pointing core.hooksPath at scripts/, which would disable the LFS hooks. Corpus data, personal data above all, stays outside the repository.

The release gateCopy Anchor Link

scripts/release_check.sh is the local gate. It stops at the first failure.

The Python suites that need ffmpeg, ssimulacra2 or cjxl skip those legs when the tools are missing.

The gate does not run cargo deny, the semver check, the Windows job, the engine-only clippy, forge-core, cargo-fuzz, tests/test_offline_guard.py, tests/test_blob_bridge.py, tests/test_space_bridge.py or the self-tests under python/forge/. Its last line suggests a tag command; releases are tagged by the maintainer with a signed tag, so a contributor can ignore it.

CICopy Anchor Link

.github/workflows/ci.yml runs on every pull request, on push to main, nightly at 03:17 UTC and on manual dispatch, with RUSTFLAGS=-D warnings.

CI and the local gate overlap but are not the same. CI builds no Python extension and runs none of the eight Python suites, only ruff. It adds deny, semver, the Windows job, fuzzing, the benches build and forge-core, which the local gate skips. A pull request needs both.

Fuzzing and miriCopy Anchor Link

A change to a section decoder or a search path should run the deterministic mutation harnesses, which also run under cargo test:

cargo test --release -p urna-format --test mutation_fuzz
cargo test --release -p urna-runtime --test mutation_fuzz
URNA_MUTATION_ITERS=25000 cargo test --release -p urna-format --test mutation_fuzz

The default is 1500 iterations for the format harness and 250 for the runtime harness. Half of the mutations reseal the checksums so the corruption reaches the decoders.

The coverage-guided targets live in the separate fuzz/ workspace and need a nightly toolchain and cargo install cargo-fuzz. The targets are urna-view, section-decoders, runtime-indexes and mmap-open-search:

cd fuzz
cargo +nightly fuzz run urna-view -- -max_total_time=600

sh scripts/fuzz_soak.sh [seconds] runs every target for an hour each by default, keeping the corpus under fuzz/corpus/<target>; URNA_FUZZ_TARGETS narrows the list. A new codec gets an arm in fuzz/fuzz_targets/section_decoders.rs. A finding becomes a crates/*/tests/negative_*.rs test and a fuzz/seeds/regress-*.bin seed before the fix.

To run miri the way CI does:

MIRIFLAGS="-Zmiri-disable-isolation -Zmiri-tree-borrows" cargo +nightly miri test -p urna-format

Tests that call zstd, the half f16 conversions on aarch64 or the FSST table build are marked to skip under miri, with the reason in the source.

Code rulesCopy Anchor Link

Rust:

• cargo fmt --all with the settings pinned in rustfmt.toml (width 100).
• cargo clippy --workspace --all-targets -- -D warnings is a hard gate. Suppress a lint on one item with #[allow(clippy::name)] and a one-line justification, never globally. clippy.toml pins cognitive complexity at 15, type complexity at 250 and at most 7 arguments.
• clippy::unwrap_used and clippy::undocumented_unsafe_blocks are denied workspace-wide. Tests may unwrap. Parse paths read fixed-width fields through urna_format::bytes, and every unsafe block carries a // SAFETY: comment that names its invariant.
• Public items get a doc comment that explains why, not what.

Python:

• Ruff with target-version = "py312", line length 100 and the lint set E F W I B UP SIM, configured in the root pyproject.toml.
• scripts/ruff_check.sh holds the one file list that the gate and CI both check. When you touch a Python module, add it to the list and make it clean.
• Private helpers in python/tools/ start with _, for example _baseline_decoder.py.

File size: 639 lines per code file. Above that, split along single-responsibility lines in the same pull request. Tests, data, generated files, lockfiles, JSON, YAML, TOML and vendored files are exempt. The guards enforce it for Rust under crates/ only; Python, forge-core and fuzz follow it by convention.

The format:

• The container is frozen at v1. A byte-level change either fits inside v1, with a new section id or encoding id that does not collide with one already emitted, or bumps URNA_FORMAT_VERSION and ships as v2. The ids in use are listed in sections and encodings.
• New manifest fields are Option with skip_serializing_if, so an unset field writes nothing and old files stay byte-identical. New capability flags go in capabilities_ext or the flattened extra map, never as a new required bool. crates/urna-format/tests/manifest_additivity.rs guards this.

Repository docs start with a YAML header (project, audience, status, last-updated, domain); the README, the license, the pull request template and llms.txt are exempt. No emoji and no em dash in text. A change to architecture, module boundaries, data flow or doc locations updates docs/arc/ARC.toml in the same pull request, and a user-visible change gets a line under [Unreleased] in docs/CHANGELOG. The pull request template lists the rest as checkboxes. A coding agent reads .contracts/.agents/AGENTS.md; the root CLAUDE.md is a link to it.

Report issuesCopy Anchor Link

• Bugs and feature requests: GitHub issues.
• Security problems: never a public issue. Follow report a vulnerability.
• Questions about the format: a GitHub discussion, or docs/arc/ARC.toml.

A bug report should carry the file_hash and content_hash of the .urna involved and the simd_backend (all three from urna stats <file>), the exact CLI or Python invocation, and the error output.

Conduct and licenseCopy Anchor Link

The project follows docs/CODE_OF_CONDUCT.md. Contributions are licensed under the MIT license; copyright vests in Hoff Research as the maintainer.

How the pieces fit together is on the architecture page.