{ "cells": [ { "cell_type": "markdown", "id": "c00", "metadata": {}, "source": [ "# A training cohort with a study-level split\n", "\n", "A model trained on spatial data should be tested on data from experiments it has not seen. If sections from the\n", "same experiment end up in both the training and the test fold, the test score measures how well the model\n", "recognises that experiment, not how well it generalises. The notebook builds a small cohort for training, splits\n", "it so that each experiment stays in one fold, reads one table per fold and records the split so that it can be\n", "reproduced.\n", "\n", "It uses the steps from the tutorials, mainly {doc}`../tutorials/03_cohorts_and_manifests`, and adds\n", "`split()` and the `TableAdapter` interface. The cohort is small (five Visium sections of human pancreatic\n", "tumours, of which only the tables are read), so it runs on a laptop." ] }, { "cell_type": "code", "execution_count": 1, "id": "c01", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:11.436598Z", "iopub.status.busy": "2026-10-31T23:40:11.436231Z", "iopub.status.idle": "2026-10-31T23:40:12.678096Z", "shell.execute_reply": "2026-10-31T23:40:12.676753Z" } }, "outputs": [], "source": [ "import tempfile\n", "from pathlib import Path\n", "from typing import Any\n", "\n", "import sddb\n", "from sddb import SplitManifest, SpatialDataCohort, TableAdapter\n", "\n", "cat = sddb.Catalog(version=\"2026-10-28\")\n", "work = Path(tempfile.mkdtemp()) # the files this notebook writes go here" ] }, { "cell_type": "markdown", "id": "c02", "metadata": {}, "source": [ "## The cohort\n", "\n", "Five Visium sections of human pancreatic ductal adenocarcinoma from three GEO series. The `collections` column\n", "names the series of each section." ] }, { "cell_type": "code", "execution_count": 2, "id": "c03", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:12.682203Z", "iopub.status.busy": "2026-10-31T23:40:12.681952Z", "iopub.status.idle": "2026-10-31T23:40:12.698488Z", "shell.execute_reply": "2026-10-31T23:40:12.697444Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
uidstudy_idcollectionsn_obsn_features
0qEP9pd4LlyKeufe20000geo:GSM8915497[geo-gse294669]114217943
10877Bxvk1AFUpoD90000geo:GSM8641038[geo-gse282302]226119144
2Mc06lamGrOTYHhNF0000geo:GSM8714181[geo-gse274557]41936601
3URKDREjVhAsRwwF10000geo:GSM8714178[geo-gse274557]66036601
4StreGVL1Uyj2EeoM0000geo:GSM8641015[geo-gse282302]230219144
\n", "
" ], "text/plain": [ " uid study_id collections n_obs n_features\n", "0 qEP9pd4LlyKeufe20000 geo:GSM8915497 [geo-gse294669] 1142 17943\n", "1 0877Bxvk1AFUpoD90000 geo:GSM8641038 [geo-gse282302] 2261 19144\n", "2 Mc06lamGrOTYHhNF0000 geo:GSM8714181 [geo-gse274557] 419 36601\n", "3 URKDREjVhAsRwwF10000 geo:GSM8714178 [geo-gse274557] 660 36601\n", "4 StreGVL1Uyj2EeoM0000 geo:GSM8641015 [geo-gse282302] 2302 19144" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cohort = cat.cohort(\n", " uids=[\n", " \"qEP9pd4LlyKeufe20000\",\n", " \"URKDREjVhAsRwwF10000\",\n", " \"Mc06lamGrOTYHhNF0000\",\n", " \"0877Bxvk1AFUpoD90000\",\n", " \"StreGVL1Uyj2EeoM0000\",\n", " ]\n", ")\n", "cohort.to_df()[[\"uid\", \"study_id\", \"collections\", \"n_obs\", \"n_features\"]]" ] }, { "cell_type": "markdown", "id": "c04", "metadata": {}, "source": [ "## Which key keeps an experiment together\n", "\n", "`split(by=...)` assigns whole groups to folds, and the default group is `study_id`. For GEO data the `study_id`\n", "is the sample accession, so every section is its own group, and sections of one series can land in different\n", "folds:" ] }, { "cell_type": "code", "execution_count": 3, "id": "c05", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:12.701657Z", "iopub.status.busy": "2026-10-31T23:40:12.701495Z", "iopub.status.idle": "2026-10-31T23:40:12.738293Z", "shell.execute_reply": "2026-10-31T23:40:12.736889Z" } }, "outputs": [ { "data": { "text/plain": [ "{'train': ['geo-gse274557', 'geo-gse294669'],\n", " 'val': ['geo-gse282302'],\n", " 'test': ['geo-gse282302']}" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def series_per_fold(folds: dict[str, SpatialDataCohort]) -> dict[str, list[str]]:\n", " \"\"\"The GEO series in each fold, from the collections column.\"\"\"\n", " return {fold: sorted({c for cs in members.members[\"collections\"] for c in cs}) for fold, members in folds.items()}\n", "\n", "\n", "by_sample = cohort.split(by=\"study_id\", train=0.6, val=0.2, test=0.2, seed=0)\n", "series_per_fold(by_sample)" ] }, { "cell_type": "markdown", "id": "c06", "metadata": {}, "source": [ "A series in two folds means that the test fold holds sections of an experiment the model was trained on.\n", "`split(by=...)` also takes a function that gets one member row and returns its group key. The function below\n", "returns the GEO series from the `collections` column, so each series stays in one fold:" ] }, { "cell_type": "code", "execution_count": 4, "id": "c07", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:12.741103Z", "iopub.status.busy": "2026-10-31T23:40:12.740951Z", "iopub.status.idle": "2026-10-31T23:40:12.752891Z", "shell.execute_reply": "2026-10-31T23:40:12.751812Z" } }, "outputs": [ { "data": { "text/plain": [ "{'train': ['geo-gse294669'],\n", " 'val': ['geo-gse274557'],\n", " 'test': ['geo-gse282302']}" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def geo_series(row) -> str:\n", " \"\"\"The GEO series of a member, from its collections.\"\"\"\n", " return next(c for c in row[\"collections\"] if c.startswith(\"geo-gse\"))\n", "\n", "\n", "folds = cohort.split(by=geo_series, train=0.4, val=0.3, test=0.3, seed=0)\n", "series_per_fold(folds)" ] }, { "cell_type": "markdown", "id": "c08", "metadata": {}, "source": [ "Fold sizes are counted in groups, not in sections, so with three series each fold gets one series, whatever the\n", "ratios. With more series the ratios apply as you would expect. For sources other than GEO, check which column\n", "groups an experiment (for example `collections` or `doi`) before you split.\n", "\n", "## A table adapter\n", "\n", "`to_anndata(adapter=...)` passes each member's table through an adapter before the join. An adapter is any object\n", "with the methods of the `TableAdapter` protocol, so it can check a member, drop it, or map its table into the\n", "space your model expects. The adapter below keeps the spots under the tissue and drops a section with fewer than 500 such\n", "spots." ] }, { "cell_type": "code", "execution_count": 5, "id": "c09", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:12.756276Z", "iopub.status.busy": "2026-10-31T23:40:12.756134Z", "iopub.status.idle": "2026-10-31T23:40:12.762005Z", "shell.execute_reply": "2026-10-31T23:40:12.760970Z" } }, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "class InTissueSpots:\n", " \"\"\"Keep spots under the tissue; drop sections with fewer than ``min_spots`` of them.\"\"\"\n", "\n", " name = \"in-tissue-spots\"\n", " version = \"0.1.0\"\n", "\n", " def __init__(self, min_spots: int = 200) -> None:\n", " self.min_spots = min_spots\n", "\n", " def requirements(self) -> dict[str, Any]:\n", " return {} # read the whole table\n", "\n", " def validate_meta(self, row) -> list[str]:\n", " return [] if row[\"technology\"] == \"Visium\" else [f\"not Visium: {row['technology']}\"]\n", "\n", " def validate(self, adata, row) -> list[str]:\n", " if \"in_tissue\" not in adata.obs:\n", " return [\"no obs['in_tissue']\"]\n", " n = int((adata.obs[\"in_tissue\"] == 1).sum())\n", " return [] if n >= self.min_spots else [f\"only {n} spots under the tissue\"]\n", "\n", " def build(self, adata, row):\n", " return adata[adata.obs[\"in_tissue\"] == 1].copy()\n", "\n", "\n", "adapter = InTissueSpots(min_spots=500)\n", "isinstance(adapter, TableAdapter)" ] }, { "cell_type": "markdown", "id": "c10", "metadata": {}, "source": [ "## One table per fold\n", "\n", "Each fold is a cohort, so `to_anndata` reads its tables and joins them. `gene_key=\"symbol\"` puts the genes of\n", "every member under their symbols." ] }, { "cell_type": "code", "execution_count": 6, "id": "c11", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:12.764831Z", "iopub.status.busy": "2026-10-31T23:40:12.764634Z", "iopub.status.idle": "2026-10-31T23:40:32.323875Z", "shell.execute_reply": "2026-10-31T23:40:32.322570Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "train 1142 spots x 17943 genes from 1 sections\n", "val 660 spots x 36601 genes from 1 sections\n", "test 4563 spots x 19144 genes from 2 sections\n" ] } ], "source": [ "tables = {fold: members.to_anndata(gene_key=\"symbol\", adapter=adapter) for fold, members in folds.items()}\n", "for fold, adata in tables.items():\n", " print(f\"{fold:5s} {adata.n_obs:5d} spots x {adata.n_vars} genes from {adata.obs['uid'].nunique()} sections\")" ] }, { "cell_type": "markdown", "id": "c12", "metadata": {}, "source": [ "Sections that the adapter dropped are listed in `uns[\"sddb\"][\"dropped\"]` with the reason:" ] }, { "cell_type": "code", "execution_count": 7, "id": "c13", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:32.327675Z", "iopub.status.busy": "2026-10-31T23:40:32.327252Z", "iopub.status.idle": "2026-10-31T23:40:32.332081Z", "shell.execute_reply": "2026-10-31T23:40:32.331165Z" } }, "outputs": [ { "data": { "text/plain": [ "{'train': {},\n", " 'val': {'Mc06lamGrOTYHhNF0000': 'only 419 spots under the tissue'},\n", " 'test': {}}" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "{fold: adata.uns.get(\"sddb\", {}).get(\"dropped\", {}) for fold, adata in tables.items()}" ] }, { "cell_type": "markdown", "id": "c14", "metadata": {}, "source": [ "The folds have different numbers of genes, because the series store different gene lists (17943, 36601 and 19144 genes). A model needs the\n", "same features in every fold, so keep the genes that all folds share, in one fixed order:" ] }, { "cell_type": "code", "execution_count": 8, "id": "c15", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:32.335596Z", "iopub.status.busy": "2026-10-31T23:40:32.335309Z", "iopub.status.idle": "2026-10-31T23:40:32.534036Z", "shell.execute_reply": "2026-10-31T23:40:32.532865Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "17943 genes in every fold\n" ] }, { "data": { "text/plain": [ "{'train': (1142, 17943), 'val': (660, 17943), 'test': (4563, 17943)}" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "shared = sorted(set.intersection(*(set(adata.var_names) for adata in tables.values())))\n", "tables = {fold: adata[:, shared].copy() for fold, adata in tables.items()}\n", "print(len(shared), \"genes in every fold\")\n", "{fold: adata.shape for fold, adata in tables.items()}" ] }, { "cell_type": "markdown", "id": "c16", "metadata": {}, "source": [ "`X` holds the counts and `obs[\"uid\"]` the section of each spot, which is what a training loop needs." ] }, { "cell_type": "code", "execution_count": 9, "id": "c17", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:32.537288Z", "iopub.status.busy": "2026-10-31T23:40:32.537140Z", "iopub.status.idle": "2026-10-31T23:40:32.548236Z", "shell.execute_reply": "2026-10-31T23:40:32.547260Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "csr_matrix (1142, 17943) 197054 counts\n" ] }, { "data": { "text/plain": [ "uid\n", "qEP9pd4LlyKeufe20000 1142\n", "Name: count, dtype: int64" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train = tables[\"train\"]\n", "print(type(train.X).__name__, train.X.shape, f\"{train.X.sum():.0f} counts\")\n", "train.obs[\"uid\"].value_counts()" ] }, { "cell_type": "markdown", "id": "c18", "metadata": {}, "source": [ "## Record the split\n", "\n", "Every fold carries the same `SplitManifest`. It records the group column, the seed, the ratios, the fold of every\n", "uid and the hash of the cohort it was made from. Its `by` records the function name. Write it next to the model,\n", "together with the cohort manifest, which records the catalog release. Keep the grouping function in your code,\n", "because the manifest records only its name." ] }, { "cell_type": "code", "execution_count": 10, "id": "c19", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:32.550867Z", "iopub.status.busy": "2026-10-31T23:40:32.550529Z", "iopub.status.idle": "2026-10-31T23:40:32.557600Z", "shell.execute_reply": "2026-10-31T23:40:32.556690Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "geo_series 0 {'test': 0.3, 'train': 0.4, 'val': 0.3}\n", "same assignments: True\n", "made from this cohort: True\n" ] } ], "source": [ "split = folds[\"train\"].split_manifest\n", "split.write(work / \"split.json\")\n", "cohort.freeze().write(work / \"cohort.json\")\n", "\n", "again = SplitManifest.load(work / \"split.json\")\n", "print(again.by, again.seed, again.ratios)\n", "print(\"same assignments:\", again.assignments == split.assignments)\n", "print(\"made from this cohort:\", again.parent_hash == cohort.freeze().hash)" ] }, { "cell_type": "markdown", "id": "c20", "metadata": {}, "source": [ "`split()` is deterministic. The same cohort, column, ratios and seed give the same folds on any machine, because\n", "the groups are sorted before a seeded shuffle." ] }, { "cell_type": "code", "execution_count": 11, "id": "c21", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:32.559909Z", "iopub.status.busy": "2026-10-31T23:40:32.559771Z", "iopub.status.idle": "2026-10-31T23:40:32.571374Z", "shell.execute_reply": "2026-10-31T23:40:32.570431Z" } }, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "repeat = cohort.split(by=geo_series, train=0.4, val=0.3, test=0.3, seed=0)\n", "repeat[\"train\"].split_manifest.assignments == split.assignments" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.15" } }, "nbformat": 4, "nbformat_minor": 5 }