sddb.loader.materialize#

sddb.loader.materialize(cohort: Iterable[Any], dest: str | Path, *, adapters: Sequence[TaskAdapter], role: Literal['he', 'if'] | None = None, roles: Mapping[str, str] | None = None, mpp_native: Mapping[str, float | None] | None = None, obs_columns: Sequence[str] = (), runner: Literal['local', 'snakemake'] = 'local', executor: Literal['local', 'slurm'] = 'local', workers: int = 1, allow_mixed_grids: bool = False, slurm: Mapping[str, Any] | None = None, stores: Mapping[str, str | Path] | None = None) → Path#

Write every member of cohort as one dataset of a loader collection at dest, and validate it.

Members are opened lazily (to_sdata) and written in uid order as dataset_0, dataset_1, ... The run state lives beside the collection, in <dest>.sddb/ (done/<uid> markers and scratch/), so that dest holds only zarr groups. Per member:

  1. its table is put on the collection's var, the union of the members' features, sorted: the outer join of to_anndata() (ad.concat(join="outer") in sddb/concat.py). The "Gene Expression" features (var["feature_types"]; all of them without that column) are re-keyed to their Ensembl id (gene_key="ensembl"), kept in var["gene_ids"]. A Gene Expression feature without an id keeps its name (gene_ids null), as do the other features, e.g. control probes; repeated names, such as two rows with one id, are made unique (A, A-1). Features join by that name. var["feature_types"], var["gene_ids"] and var["original_var_name"] (the source name, e.g. the gene symbol) are carried from the first member that has the feature. A source without var["feature_types"] has its features typed "Gene Expression", spatialdata-io's rule, and the GeneKey record says so (feature_types_from_source). A feature a member did not measure is 0 in its X, and var["present"] of its dataset is False there; the contract counts both (n_features_present, n_features_filled). On a rerun the var of the datasets already written is binding, and a member with a feature outside it raises. No feature is left out. The counts (features without an id, repeated ids, renamed, filled) go into the GeneKey record, the first of uns["sddb_contract"]["adapters"] (a step of this function, not an adapter: see sddb.loader.MATERIALIZE_RECORDS), and are logged;

  2. the adapters run in order. An adapter with a fit method and no reference (StainNormalize) is fit once on all members at that point of the chain, or takes the reference of the members already written;

  3. after the step that makes labels["cells"] (Rasterize), the labels are written once to <dest>.sddb/scratch and reread from there, so that the later steps and the final write do not draw the cells again. They are deleted once the member is validated and marked done;

  4. obs["total_counts"], each cell's sum over the "Gene Expression" features (var["feature_types"] of the collection's var, see 1.), and var["mean_counts"] and var["var_counts"], each feature's mean and variance (n - 1 degrees of freedom) over the dataset's cells, are computed in one reduction over X (block by block for a dask X: one more read of the source). A source obs["total_counts"] is kept as obs["total_counts_source"];

  5. the dataset is written with spatialbatch.write_collection_dataset() with uns["sddb_contract"] and uns["spatial"][uid] into a store of its own in <dest>.sddb/scratch, checked with validate_dataset() and moved into dest. Then <dest>.sddb/done/<uid> is written, and a rerun skips the members that have it.

Last, the root of dest is marked as an annbatch collection, the metadata of all datasets is consolidated into it, and the whole collection is checked with validate(). Only this last step writes the root metadata, so members written in parallel never race on it.

With runner="snakemake" the same steps run as a workflow, generated into <dest>.sddb/Snakefile and <dest>.sddb/config.yaml: rule fit_stain (only if the chain fits a stain reference) writes <dest>.sddb/stain.json, rule dataset_<uid> writes one member (its output is the done marker), and rule validate runs the last step and writes <dest>.sddb/valid.json. The shared var and the dataset keys are fixed in config.yaml before any job runs, and each job rebuilds the chain from the adapters' records (adapter_from_record()), so every adapter must be a loader adapter.

Memory: a member with a local store (a Dataset) is opened with its tables' X as dask arrays (_read_store), as the snakemake jobs open it, so that the controller holds only obs and var of the members not yet written. X is held in memory once, by spatialbatch.write_collection_dataset() after it sorts the rows, so a member's write holds one X in memory; the totals of step 4 hold one block at a time. Images and labels stay lazy (dask) and are written block by block.

Parameters:
cohort

The members: Dataset records, e.g. a SpatialDataCohort.

dest

The collection, a local zarr v3 group; created if missing.

adapters

The chain, e.g. default_chain(). It must hold one Rasterize and, after it, Centroids.

role

The image role of the collection, "he" or "if". By default "he" if every member has an H&E image (catalog hne_image), else "if" if every member has an IF image (if_image).

roles

Regexes for the image, the table and optionally existing labels of every member (see LoaderItem.from_sdata()). By default those of the members' technology and role, the element names its spatialdata-io reader writes (sddb.loader._defaults.ROLES; so far 10x Atera with "he"); members of several technologies, or one without defaults, need them passed.

mpp_native

Microns per scale-0 image pixel by uid, overriding the defaults; None means unknown, and the member keeps its native grid. A value given here is not checked. A member without an entry (see sddb.loader._defaults.resolve_mpp):

  • 10x_visium: the fit of its spots' pixel centres on their array positions (100 µm between neighbours of a row), never the catalog mpp, which up to release 2026-11-12 is 1.18 to 1.27 times too small. In later releases a catalog value more than 2% off the fit raises;

  • otherwise its catalog mpp, checked against the one the store records where it records one (sdata.attrs["pixel_size"] times the image's scale onto global, as spatialdata-io's Atera reader writes): a relative difference above 1e-6 raises. A null catalog value means unknown.

The contract's mpp_source says which applied.

obs_columns

Catalog columns copied into every dataset's obs, as categoricals.

stores

A local store by uid, opened in place of the member's catalog store, e.g. a copy or a derived store (HD bins aggregated into pseudo-spots). The member stays the catalog record: its catalog columns, defaults and the cohort's release go into the contract. Without an entry a remote store is fetched into the cache with only the elements the roles select (and the elements its table annotates), a local one is opened in place.

runner

"local" runs in this process; "snakemake" runs the generated workflow with snakemake.

executor

"local", or "slurm" with the snakemake runner (snakemake --executor slurm).

workers

Threads of the dask scheduler with the local runner; jobs at once with the snakemake runner (--cores locally, --jobs on Slurm), each running one dask thread.

allow_mixed_grids

Allow a collection where some members are on the target grid and others (unknown mpp) on their native grid.

slurm

Default resources of the Slurm jobs, passed as --default-resources: runtime (minutes, or a span such as "2h") and exactly one of mem_mb and mem_mb_per_cpu (an int of MB) are required; slurm_partition, slurm_account and the other resources of the snakemake Slurm executor are optional. slurm["fit_stain"] overrides them for the stain fit, which holds the tables of all members, e.g. {"mem_mb": 16000, "runtime": "2h", "fit_stain": {"mem_mb": 64000}}. Only with executor="slurm". The jobs run on the compute nodes, so dest, its run state and every member's store (an absolute path) must be on a filesystem that the nodes share with this machine.

Returns:

dest as a Path.

Raises:
  • ValueError -- On invalid arguments, a collection that would mix grids without allow_mixed_grids, a dest holding datasets of another cohort, or members that share no gene.

  • ContractError -- If a written dataset or the collection does not conform to the contract.

  • RuntimeError -- If the snakemake workflow fails; the message names its log.