API reference#

sddb.Catalog

The dataset catalog: fetched, cached and validated on construction.

sddb.Dataset

One catalog row: a published dataset.

sddb.SpatialDataCohort

An ordered, sliceable set of Dataset records from a query/search.

sddb.TaskAdapter

What a task repo implements.

sddb.TableAdapter

Per-table counterpart of TaskAdapter, used by to_anndata().

sddb.Manifest

What was downloaded, where, and from which catalog.

sddb.CohortManifest

Serializable cohort identity: filter, member snapshot (pinned full uids) and a deterministic hash.

sddb.SplitManifest

Per-uid fold assignment, pinned to the parent cohort hash, by key, seed and ratios.

sddb.CohortDriftError

A pinned cohort member no longer resolves; drifted maps uid -> reason.

sddb.CatalogChangedError

The pinned catalog's bytes differ from the sha256 recorded when the cohort was frozen.

sddb.CatalogUnavailableError

The catalog a cohort was frozen from could not be fetched, so its recorded sha256 could not be verified.

sddb.ManifestVersionMismatch

The catalog version differs from the one recorded in an existing manifest.

sddb.open_sdata

Open a SpatialData zarr from a URL or local path, anonymously.

sddb.resolve

Resolve each role to exactly one element path.

sddb.table_centroids

Centroid of each table row in an image's scale-0 pixel space.

sddb.citations.citation_entries

Return (BibTeX text, n_published, n_fallback) that cite every study in results.

Loader collections#

sddb.loader writes a cohort as a collection for the spatialbatch loader, and sddb.contract defines and checks what the collection holds. Both need the loader extra and spatialbatch, installed as in Training crops from whole Atera H&E slides (section "Install").

sddb.loader

The training-data path: adapters that bring a cohort member onto the loader contract (sddb.contract).

sddb.loader.materialize

Write every member of cohort as one dataset of a loader collection at dest, and validate it.

sddb.loader.default_chain

[Rasterize(mpp), Centroids(), TissueMask(), StainNormalize()], the H&E chain.

sddb.loader.Rasterize

Resample the image and the cells onto the written grid.

sddb.loader.Centroids

Each cell's centre of mass in labels["cells"] as obsm["spatial"] (x, y), in table row order.

sddb.loader.TissueMask

Detect tissue with squidpy.experimental.im.detect_tissue() and flag the cells on it.

sddb.loader.StainNormalize

Normalise H&E stains to one reference with squidpy's stain module.

sddb.loader.adapter_from_record

Rebuild any loader adapter from its record (an adapter's to_record()).

sddb.contract

The loader contract (version 0.1.0): what a cohort materialised for spatialbatch looks like on disk.

sddb.contract.validate

Check a collection: every dataset (see validate_dataset()) plus the checks across datasets.

sddb.contract.validate_dataset

Check one dataset of a collection against the contract.

sddb.contract.assert_valid

Raise ContractError if validate() finds issues.

sddb.contract.ContractError

A collection does not conform to the contract; issues lists why.

LoaderItem is a dataclass and the contract models are pydantic models; their fields are described in each docstring.

class sddb.loader.LoaderItem(uid: str, study_id: str, role: ~typing.Literal['he', 'if'], table: ~anndata._core.anndata.AnnData, image: ~xarray.core.dataarray.DataArray, labels: dict[str, ~xarray.core.dataarray.DataArray], mpp_native: float | None, mpp_target: float | None = None, grid: ~typing.Literal['target', 'native'] | None = None, records: list[~sddb.contract._models.AdapterRecord] = <factory>, flags: dict[str, bool] = <factory>, channels: list[str] | None = None, dtype_rule: str | None = None, stain_reference: dict[str, ~typing.Any] | None = None, shape_centroids: ~numpy.ndarray | None = None, table_centroids: ~numpy.ndarray | None = None, table_centroids_element: str | None = None, native_cells: ~xarray.core.dataarray.DataArray | None = None, native_cells_to_index: ~numpy.ndarray | None = None, n_centroid_fallback: int = 0, image_footprint: ~numpy.ndarray | None = None, n_outside_image: int = 0, label_rule: str = '', he_alpha_dropped: int | None = None, source_coarse: ~typing.Any = None, source_tissue: ~numpy.ndarray | None = None, source: ~sddb.loader._item._Source | None = None)#

One cohort member on its way through the loader adapters.

Each adapter returns a new item and appends its AdapterRecord to records. Task code writes the last item as one dataset of a loader collection.

uid, study_id

The catalog member.

role#

"he" or "if". An "he" image is 3-channel (RGB) uint8; an RGBA uint8 source with a constant alpha loses the alpha (he_alpha_dropped).

Type:

Literal['he', 'if']

table#

The cells, one row per cell of the annotated element. obs["label_id"] holds each row's value in labels["cells"] (see from_sdata()); Centroids adds obsm["spatial"].

Type:

anndata._core.anndata.AnnData

image#

(c, y, x), dask-backed. Before Rasterize it is the source image's scale 0, after it the image on the written grid.

Type:

xarray.core.dataarray.DataArray

labels#

"cells" after Rasterize, "tissue" after TissueMask, both (y, x) on the image's grid.

Type:

dict[str, xarray.core.dataarray.DataArray]

mpp_native#

Microns per scale-0 pixel of the source image, None if unknown.

Type:

float | None

mpp_target#

Microns per pixel of the written grid on the "target" grid, None on the native grid.

Type:

float | None

grid#

"target" or "native", set by Rasterize.

Type:

Literal['target', 'native'] | None

records#

The adapter steps, in the order they ran.

Type:

list[sddb.contract._models.AdapterRecord]

flags#

allow_upsample, allow_mixed_grids and upsampled.

Type:

dict[str, bool]

channels#

IF channel names in image order (from the image's c coordinate), None for H&E.

Type:

list[str] | None

dtype_rule#

How the source pixels were converted to the role's dtype, None if they already had it.

Type:

str | None

stain_reference#

The stain reference applied by StainNormalize, as plain lists.

Type:

dict[str, Any] | None

shape_centroids#

(n_obs, 2) (x, y) centroid of each row's polygon in written pixel-index coordinates, set by Rasterize when the cells are shapes (NaN for a row without a shape), None otherwise. Centroids uses it for cells that have no pixel on the written grid.

Type:

numpy.ndarray | None

table_centroids#

(n_obs, 2) the source table's obsm["spatial"] mapped into written pixel-index coordinates by Rasterize, as intrinsic coordinates of the element the table annotates; None without one. table_centroids_element names that element, for Centroids' check that the mapping agrees with the cells that have pixels.

Type:

numpy.ndarray | None

native_cells, native_cells_to_index

When the cells are a labels element: its scale 0 (y, x), and the 3x3 (x, y) affine from its intrinsic coordinates to written pixel-index coordinates (its own transformation), set by Rasterize. Centroids takes a cell's centre of mass there when the cell has no pixel on the written grid. None for shape cells.

n_centroid_fallback#

How many rows took their centroid from native_cells, shape_centroids or table_centroids, set by Centroids.

Type:

int

image_footprint#

(4, 2) (x, y) corners of the source image's scale-0 extent in written pixel-index coordinates, set by Rasterize. The written grid is the bounding box of this quadrilateral, so with a rotated image its corners are fill. Centroids flags cells whose centroid is not on it.

Type:

numpy.ndarray | None

n_outside_image#

How many rows have obs["outside_image"] True, set by Centroids.

Type:

int

source_coarse#

The source image's coarsest level in the role's dtype, its index and its map from the written grid, set by Rasterize. squidpy's tissue detection and stain statistics run on it.

Type:

Any

source_tissue#

The tissue mask on source_coarse, set by TissueMask.

Type:

numpy.ndarray | None

label_rule#

Where obs["label_id"] came from (see from_sdata()).

Type:

str

he_alpha_dropped#

For an RGBA H&E source, the constant alpha value that from_sdata() dropped, None otherwise.

Type:

int | None

classmethod from_sdata(sdata: SpatialData, *, uid: str, study_id: str, role: Literal['he', 'if'], roles: Mapping[str, str], mpp_native: float | None, label_column: str | None = None) → LoaderItem#

Resolve the image, the table and the cells of one member.

Parameters:
sdata

The member, opened lazily.

uid

See the attributes.

study_id

See the attributes.

role

See the attributes.

roles

Regexes (see sddb.resolve()) for "image" and "table", and optionally "labels": an existing cell raster to resample instead of rasterising the shapes the table annotates.

mpp_native

Microns per scale-0 pixel of the image, None if unknown (the item then keeps its native grid).

label_column

The obs column holding each row's value in the cell raster; by default it is inferred, see below.

Notes

obs["label_id"], the value of each row's cell in labels["cells"], is taken from the first that applies:

  1. label_column;

  2. the instance ids, if the cells are the labels element the table annotates;

  3. obs["label_id"], else obs["cell_labels"]: the spatialdata-io conventions when the labels values differ from the instance ids (Atera: the table annotates cell_boundaries by cell_id, cell_labels holds label_id; Xenium read with cells_as_circles=True: the table annotates cell_circles by cell_id and obs["cell_labels"] holds each row's value in cell_labels). With existing labels it is checked on a sample: the labels at 200 rows' shape centroids, or, without shapes, the values of a central 512 x 512 window, must match for at least 95%;

  4. with existing labels nothing else applies, so a ValueError asks for label_column;

  5. the instance ids, if they are positive integers that fit uint32 (shapes are rasterised with them);

  6. the row position plus one (shapes are rasterised with it), e.g. for Xenium's string cell ids.

The ids must be unique, positive and fit uint32. The table keeps X, obs and var; its obsm (often in microns) and uns are dropped, as the written dataset gets its own. obsm["spatial"] is kept on the source, for Centroids' fallback for cells without pixels.

Raises:

ValueError -- If a role is unknown or names the wrong kind of element, the table does not annotate exactly one labels or shapes element, the image and the cells share no coordinate system, the label ids cannot be inferred or are invalid, or role is "he" and the image is neither 3 channels of uint8 (an IF slide the catalog marks with hne_image, for example) nor 4 of uint8 whose alpha is constant on a downsampled level.

class sddb.contract.DatasetContract(*, version: str, uid: str, study_id: str, catalog_release: str, role: Literal['he', 'if'], grid: Literal['target', 'native'], mpp_target: Annotated[float, Gt(gt=0)] | None = None, mpp_native: Annotated[float, Gt(gt=0)] | None = None, upsampled: bool, allow_upsample: bool, allow_mixed_grids: bool, dtype_rule: str | None = None, channels: list[str] | None = None, adapters: list[AdapterRecord], stain_reference: dict[str, list[float] | list[list[float]] | str] | None = None, n_centroid_fallback: Annotated[int, Ge(ge=0), Strict(strict=True)] = 0, n_outside_image: Annotated[int, Ge(ge=0), Strict(strict=True)] = 0, n_features_present: Annotated[int, Ge(ge=0), Strict(strict=True)] = 0, n_features_filled: Annotated[int, Ge(ge=0), Strict(strict=True)] = 0, totals: bool = False, mpp_source: Literal['given', 'catalog_checked_store', 'catalog_unchecked', 'store_pitch_fit'] | None = None, he_alpha_dropped: Annotated[Annotated[int, Strict(strict=True)] | None, Ge(ge=0), Le(le=255)] = None)#

uns["sddb_contract"] of a dataset: the contract version, its source and how it was written.

version#

The contract version, semver. On 0.x a breaking change bumps the minor version.

Type:

str

uid, study_id, catalog_release

The source catalog member and the catalog release it was read from.

role#

The image role of the collection: "he" (uint8, 3 channels) or "if" (uint16, one per channels).

Type:

Literal['he', 'if']

grid#

"target" if the image was resampled to mpp_target, "native" if it keeps its own grid (mpp unknown).

Type:

Literal['target', 'native']

mpp_target, mpp_native

Microns per pixel of the target grid and of the source image, None if unknown. The target grid needs both, so that upsampling can be checked.

upsampled, allow_upsample

Whether the target grid is finer than the native one, and whether the user allowed that.

allow_mixed_grids#

Whether this dataset may share a collection with datasets on the other kind of grid.

Type:

bool

dtype_rule#

How the source pixels were converted to the role's dtype, None if they already had it.

Type:

str | None

channels#

IF channel names, in image order.

Type:

list[str] | None

adapters#

The adapter steps, in the order they ran.

Type:

list[sddb.contract._models.AdapterRecord]

stain_reference#

The H&E stain normalisation reference (stain matrix, white point, max concentrations), if one was applied.

Type:

dict[str, list[float] | list[list[float]] | str] | None

n_centroid_fallback#

How many cells have no pixel in labels/cells (smaller than a pixel of the written grid), so that Centroids took their obsm["spatial"] from the source: the centre of mass in the source labels at native resolution, the polygon centroid or the source table's centroid. Optional: 0 when the key is absent.

Type:

int

n_outside_image#

How many cells have obs["outside_image"] True: their centroid is not on the H&E or IF image's footprint (beyond the written grid, or on the fill of a rotated image's corners). Their obsm["spatial"] is still their true position, so it may lie outside [0, w) x [0, h). Optional: 0 when the key is absent.

Type:

int

n_features_present, n_features_filled

How many features of the collection's var this dataset's source measured (var["present"] True), and how many it did not, which are 0 in X. Optional: 0 when the key is absent.

totals#

Whether the dataset carries the totals that materialize computes: obs["total_counts"], each cell's sum of X over the "Gene Expression" features, and var["mean_counts"] and var["var_counts"], each feature's mean and variance (n - 1 degrees of freedom) over the dataset's cells. The validator checks them only then: an obs["total_counts"] of a dataset without the flag is the source's own. Optional: False when the key is absent.

Type:

bool

mpp_source#

Where mpp_native came from: "given" (passed to materialize), "catalog_checked_store" (the catalog's mpp, which agrees with the pixel size the store records), "catalog_unchecked" (the catalog's, from a store that records none), "store_pitch_fit" (fitted on the Visium array pitch of the store's own spots, not the catalog's). None when mpp_native is unknown, or for a dataset written before the field.

Type:

Literal['given', 'catalog_checked_store', 'catalog_unchecked', 'store_pitch_fit'] | None

he_alpha_dropped#

For role "he" from an RGBA uint8 source, the alpha value that was dropped: it was constant on a downsampled level of the source. None for an RGB source, an IF dataset, or one written before the field.

Type:

int | None

class sddb.contract.SpatialEntry(*, scalefactors: Scalefactors, metadata: SpatialMetadata)#

uns["spatial"][uid], Squidpy's layout without the images arrays (the image is images/<role>).

class sddb.contract.AdapterRecord(*, name: str, version: str, params: dict[str, Any])#

One adapter step that produced a dataset, with the parameters it ran with.

class sddb.contract.Scalefactors(*, mpp: Annotated[float, Gt(gt=0)] | None, mpp_native: Annotated[float, Gt(gt=0)] | None, native_to_written: Annotated[float, Gt(gt=0)])#

uns["spatial"][uid]["scalefactors"]: the resolution of the written grid.

class sddb.contract.SpatialMetadata(*, role: Literal['he', 'if'], image: str, labels: Literal['cells'], tissue: str | None = None)#

uns["spatial"][uid]["metadata"]: the names of the dataset's elements.