Formats¶

siglip2_toolkit.formats ¶

Embeddings export format (v2): specification and read/write helpers.

An export file is a sequence of objects, optionally preceded by a header::

{"format": "siglip2-export", "version": 2,
 "embedding_space": {"model": "...", "dimensions": 768, "normalization": "l2"},
 "created_at": "2026-10-17T12:00:00+00:00"}
{"id": "/photos/a.jpg", "vector": [...], "metadata": {...}}
{"id": "/photos/b.jpg", "vector": [...]}
...

jsonl files store one object per line; pkl files store a sequence of pickled objects. Record-only files (written before format v2) are still readable.

A record carries its identity in id (any stable string; the exporter uses the absolute image path). metadata is optional and free-form: store EXIF, timestamps, tags — or nothing at all. Records without an id fall back to metadata["path"] when read back.

This is an interchange format, not a database: consumers load the vectors with whatever tool they prefer (a vector store, numpy, another pipeline).

EmbeddingSpace `dataclass` ¶

EmbeddingSpace(
    model,
    dimensions=None,
    normalization="l2",
    revision=None,
    preprocess=None,
)

Identity of the vector space produced by an embedder.

Two vectors are only comparable when they were produced by the same model (and the same preprocessing). This manifest is stored in the export header so consumers can detect mismatches.

id `property` ¶

id

Stable short identifier derived from the manifest contents.

compatible_with ¶

compatible_with(other)

Whether vectors from both spaces can be compared.

Unknown dimensions on either side are tolerated; a different model or a different declared dimension is not.

now_iso ¶

now_iso()

Current UTC time as an ISO-8601 string.

make_header ¶

make_header(embedding_space=None, created_at=None)

Build the export header object.

is_header ¶

is_header(obj)

Whether an object read from an export file is a format header.

make_record ¶

make_record(vector, metadata=None, *, id=None)

Build an export record.

id identifies the vector (enable it for incremental updates and deduplication); metadata is optional and may hold any JSON-serializable data.

record_id ¶

record_id(record)

The record id: top-level id, or legacy metadata["path"].

infer_format ¶

infer_format(path)

Infer the export format (jsonl or pkl) from a file name.

Temporary suffixes (.tmp/.part) are stripped first, so the interrupted-run convention <export>.jsonl.tmp also resolves.

read_header ¶

read_header(path)

Read the export header, or None for legacy or missing files.

iter_records ¶

iter_records(path)

Yield export records, skipping the header if present.

load_records ¶

load_records(path)

Load all records of an export file into memory.

load_by_id ¶

load_by_id(path)

Load an export file into a dictionary keyed by record id.

Records without an id (neither id nor a legacy metadata.path) are skipped with a warning. Later duplicates win, matching the behaviour of incremental runs.

write_export ¶

write_export(path, records, embedding_space=None, fmt=None)

Write records (with a header) to a new export file.

The file is created (or fully rewritten); parent directories are created as needed. A format header is always written; pass embedding_space=None when the space is unknown (e.g. when merging legacy files).

header_embedding_space ¶

header_embedding_space(header)

Extract the embedding space manifest from a header, if present.