API reference#
The dataset catalog: fetched, cached and validated on construction. |
|
One catalog row: a published dataset. |
|
An ordered, sliceable set of Dataset records from a query/search. |
|
What a task repo implements. |
|
Per-table counterpart of |
|
What was downloaded, where, and from which catalog. |
|
Serializable cohort identity: filter, member snapshot (pinned full uids) and a deterministic hash. |
|
Per-uid fold assignment, pinned to the parent cohort hash, |
|
A pinned cohort member no longer resolves; |
|
The pinned catalog's bytes differ from the sha256 recorded when the cohort was frozen. |
|
The catalog a cohort was frozen from could not be fetched, so its recorded sha256 could not be verified. |
|
The catalog version differs from the one recorded in an existing manifest. |
|
Open a SpatialData zarr from a URL or local path, anonymously. |
|
Resolve each role to exactly one element path. |
|
Centroid of each table row in an image's scale-0 pixel space. |
|
Return |
Loader collections#
sddb.loader writes a cohort as a collection for the spatialbatch loader, and sddb.contract defines and
checks what the collection holds. Both need the loader extra and spatialbatch, installed as in
Training crops from whole Atera H&E slides (section "Install").
The training-data path: adapters that bring a cohort member onto the loader contract ( |
|
Write every member of |
|
|
|
Resample the image and the cells onto the written grid. |
|
Each cell's centre of mass in |
|
Detect tissue with |
|
Normalise H&E stains to one reference with squidpy's stain module. |
|
Rebuild any loader adapter from its record (an adapter's |
|
The loader contract (version 0.1.0): what a cohort materialised for |
|
Check a collection: every dataset (see |
|
Check one dataset of a collection against the contract. |
|
Raise |
|
A collection does not conform to the contract; |
LoaderItem is a dataclass and the contract models are pydantic models; their fields are
described in each docstring.
- class sddb.loader.LoaderItem(uid: str, study_id: str, role: ~typing.Literal['he', 'if'], table: ~anndata._core.anndata.AnnData, image: ~xarray.core.dataarray.DataArray, labels: dict[str, ~xarray.core.dataarray.DataArray], mpp_native: float | None, mpp_target: float | None = None, grid: ~typing.Literal['target', 'native'] | None = None, records: list[~sddb.contract._models.AdapterRecord] = <factory>, flags: dict[str, bool] = <factory>, channels: list[str] | None = None, dtype_rule: str | None = None, stain_reference: dict[str, ~typing.Any] | None = None, shape_centroids: ~numpy.ndarray | None = None, table_centroids: ~numpy.ndarray | None = None, table_centroids_element: str | None = None, native_cells: ~xarray.core.dataarray.DataArray | None = None, native_cells_to_index: ~numpy.ndarray | None = None, n_centroid_fallback: int = 0, image_footprint: ~numpy.ndarray | None = None, n_outside_image: int = 0, label_rule: str = '', he_alpha_dropped: int | None = None, source_coarse: ~typing.Any = None, source_tissue: ~numpy.ndarray | None = None, source: ~sddb.loader._item._Source | None = None)#
One cohort member on its way through the loader adapters.
Each adapter returns a new item and appends its
AdapterRecordtorecords. Task code writes the last item as one dataset of a loader collection.- uid, study_id
The catalog member.
- role#
"he"or"if". An"he"image is 3-channel (RGB) uint8; an RGBA uint8 source with a constant alpha loses the alpha (he_alpha_dropped).- Type:
Literal['he', 'if']
- table#
The cells, one row per cell of the annotated element.
obs["label_id"]holds each row's value inlabels["cells"](seefrom_sdata());Centroidsaddsobsm["spatial"].
- image#
(c, y, x), dask-backed. BeforeRasterizeit is the source image's scale 0, after it the image on the written grid.- Type:
xarray.core.dataarray.DataArray
- labels#
"cells"afterRasterize,"tissue"afterTissueMask, both(y, x)on the image's grid.
- mpp_target#
Microns per pixel of the written grid on the
"target"grid,Noneon the native grid.- Type:
float | None
- records#
The adapter steps, in the order they ran.
- channels#
IF channel names in image order (from the image's
ccoordinate),Nonefor H&E.
- dtype_rule#
How the source pixels were converted to the role's dtype,
Noneif they already had it.- Type:
str | None
- stain_reference#
The stain reference applied by
StainNormalize, as plain lists.
- shape_centroids#
(n_obs, 2)(x, y)centroid of each row's polygon in written pixel-index coordinates, set byRasterizewhen the cells are shapes (NaN for a row without a shape),Noneotherwise.Centroidsuses it for cells that have no pixel on the written grid.- Type:
numpy.ndarray | None
- table_centroids#
(n_obs, 2)the source table'sobsm["spatial"]mapped into written pixel-index coordinates byRasterize, as intrinsic coordinates of the element the table annotates;Nonewithout one.table_centroids_elementnames that element, forCentroids' check that the mapping agrees with the cells that have pixels.- Type:
numpy.ndarray | None
- native_cells, native_cells_to_index
When the cells are a labels element: its scale 0
(y, x), and the 3x3(x, y)affine from its intrinsic coordinates to written pixel-index coordinates (its own transformation), set byRasterize.Centroidstakes a cell's centre of mass there when the cell has no pixel on the written grid.Nonefor shape cells.
- n_centroid_fallback#
How many rows took their centroid from
native_cells,shape_centroidsortable_centroids, set byCentroids.- Type:
- image_footprint#
(4, 2)(x, y)corners of the source image's scale-0 extent in written pixel-index coordinates, set byRasterize. The written grid is the bounding box of this quadrilateral, so with a rotated image its corners are fill.Centroidsflags cells whose centroid is not on it.- Type:
numpy.ndarray | None
- source_coarse#
The source image's coarsest level in the role's dtype, its index and its map from the written grid, set by
Rasterize. squidpy's tissue detection and stain statistics run on it.- Type:
Any
- source_tissue#
The tissue mask on
source_coarse, set byTissueMask.- Type:
numpy.ndarray | None
- label_rule#
Where
obs["label_id"]came from (seefrom_sdata()).- Type:
- he_alpha_dropped#
For an RGBA H&E source, the constant alpha value that
from_sdata()dropped,Noneotherwise.- Type:
int | None
- classmethod from_sdata(sdata: SpatialData, *, uid: str, study_id: str, role: Literal['he', 'if'], roles: Mapping[str, str], mpp_native: float | None, label_column: str | None = None) LoaderItem#
Resolve the image, the table and the cells of one member.
- Parameters:
- sdata
The member, opened lazily.
- uid
See the attributes.
- study_id
See the attributes.
- role
See the attributes.
- roles
Regexes (see
sddb.resolve()) for"image"and"table", and optionally"labels": an existing cell raster to resample instead of rasterising the shapes the table annotates.- mpp_native
Microns per scale-0 pixel of the image,
Noneif unknown (the item then keeps its native grid).- label_column
The
obscolumn holding each row's value in the cell raster; by default it is inferred, see below.
Notes
obs["label_id"], the value of each row's cell inlabels["cells"], is taken from the first that applies:label_column;the instance ids, if the cells are the labels element the table annotates;
obs["label_id"], elseobs["cell_labels"]: the spatialdata-io conventions when the labels values differ from the instance ids (Atera: the table annotatescell_boundariesbycell_id,cell_labelsholdslabel_id; Xenium read withcells_as_circles=True: the table annotatescell_circlesbycell_idandobs["cell_labels"]holds each row's value incell_labels). With existing labels it is checked on a sample: the labels at 200 rows' shape centroids, or, without shapes, the values of a central 512 x 512 window, must match for at least 95%;with existing labels nothing else applies, so a
ValueErrorasks forlabel_column;the instance ids, if they are positive integers that fit uint32 (shapes are rasterised with them);
the row position plus one (shapes are rasterised with it), e.g. for Xenium's string cell ids.
The ids must be unique, positive and fit uint32. The table keeps
X,obsandvar; itsobsm(often in microns) andunsare dropped, as the written dataset gets its own.obsm["spatial"]is kept on the source, forCentroids' fallback for cells without pixels.- Raises:
ValueError -- If a role is unknown or names the wrong kind of element, the table does not annotate exactly one labels or shapes element, the image and the cells share no coordinate system, the label ids cannot be inferred or are invalid, or
roleis"he"and the image is neither 3 channels of uint8 (an IF slide the catalog marks withhne_image, for example) nor 4 of uint8 whose alpha is constant on a downsampled level.
- class sddb.contract.DatasetContract(*, version: str, uid: str, study_id: str, catalog_release: str, role: Literal['he', 'if'], grid: Literal['target', 'native'], mpp_target: Annotated[float, Gt(gt=0)] | None = None, mpp_native: Annotated[float, Gt(gt=0)] | None = None, upsampled: bool, allow_upsample: bool, allow_mixed_grids: bool, dtype_rule: str | None = None, channels: list[str] | None = None, adapters: list[AdapterRecord], stain_reference: dict[str, list[float] | list[list[float]] | str] | None = None, n_centroid_fallback: Annotated[int, Ge(ge=0), Strict(strict=True)] = 0, n_outside_image: Annotated[int, Ge(ge=0), Strict(strict=True)] = 0, n_features_present: Annotated[int, Ge(ge=0), Strict(strict=True)] = 0, n_features_filled: Annotated[int, Ge(ge=0), Strict(strict=True)] = 0, totals: bool = False, mpp_source: Literal['given', 'catalog_checked_store', 'catalog_unchecked', 'store_pitch_fit'] | None = None, he_alpha_dropped: Annotated[Annotated[int, Strict(strict=True)] | None, Ge(ge=0), Le(le=255)] = None)#
uns["sddb_contract"]of a dataset: the contract version, its source and how it was written.- uid, study_id, catalog_release
The source catalog member and the catalog release it was read from.
- role#
The image role of the collection:
"he"(uint8, 3 channels) or"if"(uint16, one perchannels).- Type:
Literal['he', 'if']
- grid#
"target"if the image was resampled tompp_target,"native"if it keeps its own grid (mpp unknown).- Type:
Literal['target', 'native']
- mpp_target, mpp_native
Microns per pixel of the target grid and of the source image,
Noneif unknown. The target grid needs both, so that upsampling can be checked.
- upsampled, allow_upsample
Whether the target grid is finer than the native one, and whether the user allowed that.
- allow_mixed_grids#
Whether this dataset may share a collection with datasets on the other kind of grid.
- Type:
- dtype_rule#
How the source pixels were converted to the role's dtype,
Noneif they already had it.- Type:
str | None
- adapters#
The adapter steps, in the order they ran.
- stain_reference#
The H&E stain normalisation reference (stain matrix, white point, max concentrations), if one was applied.
- n_centroid_fallback#
How many cells have no pixel in
labels/cells(smaller than a pixel of the written grid), so thatCentroidstook theirobsm["spatial"]from the source: the centre of mass in the source labels at native resolution, the polygon centroid or the source table's centroid. Optional: 0 when the key is absent.- Type:
- n_outside_image#
How many cells have
obs["outside_image"]True: their centroid is not on the H&E or IF image's footprint (beyond the written grid, or on the fill of a rotated image's corners). Theirobsm["spatial"]is still their true position, so it may lie outside[0, w) x [0, h). Optional: 0 when the key is absent.- Type:
- n_features_present, n_features_filled
How many features of the collection's var this dataset's source measured (
var["present"]True), and how many it did not, which are 0 inX. Optional: 0 when the key is absent.
- totals#
Whether the dataset carries the totals that materialize computes:
obs["total_counts"], each cell's sum ofXover the"Gene Expression"features, andvar["mean_counts"]andvar["var_counts"], each feature's mean and variance (n - 1degrees of freedom) over the dataset's cells. The validator checks them only then: anobs["total_counts"]of a dataset without the flag is the source's own. Optional: False when the key is absent.- Type:
- mpp_source#
Where
mpp_nativecame from:"given"(passed to materialize),"catalog_checked_store"(the catalog'smpp, which agrees with the pixel size the store records),"catalog_unchecked"(the catalog's, from a store that records none),"store_pitch_fit"(fitted on the Visium array pitch of the store's own spots, not the catalog's).Nonewhenmpp_nativeis unknown, or for a dataset written before the field.- Type:
Literal['given', 'catalog_checked_store', 'catalog_unchecked', 'store_pitch_fit'] | None
- class sddb.contract.SpatialEntry(*, scalefactors: Scalefactors, metadata: SpatialMetadata)#
uns["spatial"][uid], Squidpy's layout without theimagesarrays (the image isimages/<role>).
- class sddb.contract.AdapterRecord(*, name: str, version: str, params: dict[str, Any])#
One adapter step that produced a dataset, with the parameters it ran with.