sddb.loader.materialize#
- sddb.loader.materialize(cohort: Iterable[Any], dest: str | Path, *, adapters: Sequence[TaskAdapter], role: Literal['he', 'if'] | None = None, roles: Mapping[str, str] | None = None, mpp_native: Mapping[str, float | None] | None = None, obs_columns: Sequence[str] = (), runner: Literal['local', 'snakemake'] = 'local', executor: Literal['local', 'slurm'] = 'local', workers: int = 1, allow_mixed_grids: bool = False, slurm: Mapping[str, Any] | None = None, stores: Mapping[str, str | Path] | None = None) Path#
Write every member of
cohortas one dataset of a loader collection atdest, and validate it.Members are opened lazily (
to_sdata) and written in uid order asdataset_0,dataset_1, ... The run state lives beside the collection, in<dest>.sddb/(done/<uid>markers andscratch/), so thatdestholds only zarr groups. Per member:its table is put on the collection's var, the union of the members' features, sorted: the outer join of
to_anndata()(ad.concat(join="outer")insddb/concat.py). The"Gene Expression"features (var["feature_types"]; all of them without that column) are re-keyed to their Ensembl id (gene_key="ensembl"), kept invar["gene_ids"]. A Gene Expression feature without an id keeps its name (gene_idsnull), as do the other features, e.g. control probes; repeated names, such as two rows with one id, are made unique (A,A-1). Features join by that name.var["feature_types"],var["gene_ids"]andvar["original_var_name"](the source name, e.g. the gene symbol) are carried from the first member that has the feature. A source withoutvar["feature_types"]has its features typed"Gene Expression", spatialdata-io's rule, and the GeneKey record says so (feature_types_from_source). A feature a member did not measure is 0 in itsX, andvar["present"]of its dataset is False there; the contract counts both (n_features_present,n_features_filled). On a rerun the var of the datasets already written is binding, and a member with a feature outside it raises. No feature is left out. The counts (features without an id, repeated ids, renamed, filled) go into theGeneKeyrecord, the first ofuns["sddb_contract"]["adapters"](a step of this function, not an adapter: seesddb.loader.MATERIALIZE_RECORDS), and are logged;the adapters run in order. An adapter with a
fitmethod and noreference(StainNormalize) is fit once on all members at that point of the chain, or takes the reference of the members already written;after the step that makes
labels["cells"](Rasterize), the labels are written once to<dest>.sddb/scratchand reread from there, so that the later steps and the final write do not draw the cells again. They are deleted once the member is validated and marked done;obs["total_counts"], each cell's sum over the"Gene Expression"features (var["feature_types"]of the collection's var, see 1.), andvar["mean_counts"]andvar["var_counts"], each feature's mean and variance (n - 1degrees of freedom) over the dataset's cells, are computed in one reduction overX(block by block for a daskX: one more read of the source). A sourceobs["total_counts"]is kept asobs["total_counts_source"];the dataset is written with
spatialbatch.write_collection_dataset()withuns["sddb_contract"]anduns["spatial"][uid]into a store of its own in<dest>.sddb/scratch, checked withvalidate_dataset()and moved intodest. Then<dest>.sddb/done/<uid>is written, and a rerun skips the members that have it.
Last, the root of
destis marked as an annbatch collection, the metadata of all datasets is consolidated into it, and the whole collection is checked withvalidate(). Only this last step writes the root metadata, so members written in parallel never race on it.With
runner="snakemake"the same steps run as a workflow, generated into<dest>.sddb/Snakefileand<dest>.sddb/config.yaml: rulefit_stain(only if the chain fits a stain reference) writes<dest>.sddb/stain.json, ruledataset_<uid>writes one member (its output is the done marker), and rulevalidateruns the last step and writes<dest>.sddb/valid.json. The shared var and the dataset keys are fixed inconfig.yamlbefore any job runs, and each job rebuilds the chain from the adapters' records (adapter_from_record()), so every adapter must be a loader adapter.Memory: a member with a local store (a
Dataset) is opened with its tables'Xas dask arrays (_read_store), as the snakemake jobs open it, so that the controller holds only obs and var of the members not yet written.Xis held in memory once, byspatialbatch.write_collection_dataset()after it sorts the rows, so a member's write holds oneXin memory; the totals of step 4 hold one block at a time. Images and labels stay lazy (dask) and are written block by block.- Parameters:
- cohort
The members:
Datasetrecords, e.g. aSpatialDataCohort.- dest
The collection, a local zarr v3 group; created if missing.
- adapters
The chain, e.g.
default_chain(). It must hold oneRasterizeand, after it,Centroids.- role
The image role of the collection,
"he"or"if". By default"he"if every member has an H&E image (cataloghne_image), else"if"if every member has an IF image (if_image).- roles
Regexes for the image, the table and optionally existing labels of every member (see
LoaderItem.from_sdata()). By default those of the members' technology androle, the element names its spatialdata-io reader writes (sddb.loader._defaults.ROLES; so far 10x Atera with"he"); members of several technologies, or one without defaults, need them passed.- mpp_native
Microns per scale-0 image pixel by uid, overriding the defaults;
Nonemeans unknown, and the member keeps its native grid. A value given here is not checked. A member without an entry (seesddb.loader._defaults.resolve_mpp):10x_visium: the fit of its spots' pixel centres on their array positions (100 µm between neighbours of a row), never the catalogmpp, which up to release 2026-11-12 is 1.18 to 1.27 times too small. In later releases a catalog value more than 2% off the fit raises;otherwise its catalog
mpp, checked against the one the store records where it records one (sdata.attrs["pixel_size"]times the image's scale ontoglobal, as spatialdata-io's Atera reader writes): a relative difference above 1e-6 raises. A null catalog value means unknown.
The contract's
mpp_sourcesays which applied.- obs_columns
Catalog columns copied into every dataset's
obs, as categoricals.- stores
A local store by uid, opened in place of the member's catalog store, e.g. a copy or a derived store (HD bins aggregated into pseudo-spots). The member stays the catalog record: its catalog columns, defaults and the cohort's release go into the contract. Without an entry a remote store is fetched into the cache with only the elements the roles select (and the elements its table annotates), a local one is opened in place.
- runner
"local"runs in this process;"snakemake"runs the generated workflow with snakemake.- executor
"local", or"slurm"with the snakemake runner (snakemake --executor slurm).- workers
Threads of the dask scheduler with the local runner; jobs at once with the snakemake runner (
--coreslocally,--jobson Slurm), each running one dask thread.- allow_mixed_grids
Allow a collection where some members are on the target grid and others (unknown mpp) on their native grid.
- slurm
Default resources of the Slurm jobs, passed as
--default-resources:runtime(minutes, or a span such as"2h") and exactly one ofmem_mbandmem_mb_per_cpu(anintof MB) are required;slurm_partition,slurm_accountand the other resources of the snakemake Slurm executor are optional.slurm["fit_stain"]overrides them for the stain fit, which holds the tables of all members, e.g.{"mem_mb": 16000, "runtime": "2h", "fit_stain": {"mem_mb": 64000}}. Only withexecutor="slurm". The jobs run on the compute nodes, sodest, its run state and every member's store (an absolute path) must be on a filesystem that the nodes share with this machine.
- Returns:
destas aPath.- Raises:
ValueError -- On invalid arguments, a collection that would mix grids without
allow_mixed_grids, adestholding datasets of another cohort, or members that share no gene.ContractError -- If a written dataset or the collection does not conform to the contract.
RuntimeError -- If the snakemake workflow fails; the message names its log.