{ "cells": [ { "cell_type": "markdown", "id": "c00", "metadata": {}, "source": [ "# 4. Citations and licenses\n", "\n", "Every dataset in the catalog comes from a study that someone else ran, and it keeps the license under which they\n", "shared it. Before you publish a result, you need to know which studies to cite and whether their licenses allow\n", "what you did with the data. The client records both in the catalog, and the sections below show how to read them,\n", "filter on them and write a bibliography for a cohort.\n", "\n", "You will learn how to:\n", "\n", "- read the license columns and their values\n", "- filter a query by license, and why the default does not\n", "- write BibTeX for a cohort in Python and from the shell\n", "- tell published entries from fallback entries, and fix both before you publish\n", "\n", "The tutorial downloads only the catalog and the bibliography file next to it, about 10 MB in total." ] }, { "cell_type": "code", "execution_count": 1, "id": "c01", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:04.225892Z", "iopub.status.busy": "2026-10-31T23:40:04.225487Z", "iopub.status.idle": "2026-10-31T23:40:05.469072Z", "shell.execute_reply": "2026-10-31T23:40:05.468114Z" } }, "outputs": [], "source": [ "import re\n", "import subprocess\n", "import tempfile\n", "import warnings\n", "from pathlib import Path\n", "\n", "import pandas as pd\n", "\n", "import sddb\n", "from sddb.citations import citation_entries, parse_bibtex\n", "\n", "pd.set_option(\"display.max_colwidth\", 80)\n", "\n", "cat = sddb.Catalog(version=\"2026-10-28\")\n", "df = cat.to_df()\n", "work = Path(tempfile.mkdtemp()) # the files this tutorial writes go here" ] }, { "cell_type": "markdown", "id": "c02", "metadata": {}, "source": [ "## The license columns\n", "\n", "Five columns describe the license of a dataset:\n", "\n", "- `license` is an identifier such as `CC-BY-4.0`, or `unknown`.\n", "- `license_url` links the license text.\n", "- `license_noncommercial` is true when the license forbids commercial use.\n", "- `license_redistributable` is true when the license allows you to share the data further.\n", "- `license_unknown` is true when no license could be determined.\n", "\n", "Every value in the 2026-10-28 release, with its number of datasets and the three flags:" ] }, { "cell_type": "code", "execution_count": 2, "id": "c03", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:05.472161Z", "iopub.status.busy": "2026-10-31T23:40:05.471930Z", "iopub.status.idle": "2026-10-31T23:40:05.689142Z", "shell.execute_reply": "2026-10-31T23:40:05.687616Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
datasetsnoncommercialredistributableunknownurl
license
ncbi-geo-policy3530FalseTrueFalsehttps://www.ncbi.nlm.nih.gov/geo/info/disclaimer.html
CC-BY-NC-SA-4.01072TrueTrueFalsehttps://creativecommons.org/licenses/by-nc-sa/4.0/
unknown1031FalseFalseTrue<NA>
CC-BY-4.0333FalseTrueFalsehttps://creativecommons.org/licenses/by/4.0/
allen-institute-tou115TrueTrueFalsehttps://alleninstitute.org/terms-of-use/
CC0-1.06FalseTrueFalsehttps://creativecommons.org/publicdomain/zero/1.0/
\n", "
" ], "text/plain": [ " datasets noncommercial redistributable unknown \\\n", "license \n", "ncbi-geo-policy 3530 False True False \n", "CC-BY-NC-SA-4.0 1072 True True False \n", "unknown 1031 False False True \n", "CC-BY-4.0 333 False True False \n", "allen-institute-tou 115 True True False \n", "CC0-1.0 6 False True False \n", "\n", " url \n", "license \n", "ncbi-geo-policy https://www.ncbi.nlm.nih.gov/geo/info/disclaimer.html \n", "CC-BY-NC-SA-4.0 https://creativecommons.org/licenses/by-nc-sa/4.0/ \n", "unknown \n", "CC-BY-4.0 https://creativecommons.org/licenses/by/4.0/ \n", "allen-institute-tou https://alleninstitute.org/terms-of-use/ \n", "CC0-1.0 https://creativecommons.org/publicdomain/zero/1.0/ " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "licenses = (\n", " df.groupby(\"license\")\n", " .agg(\n", " datasets=(\"uid\", \"size\"),\n", " noncommercial=(\"license_noncommercial\", \"first\"),\n", " redistributable=(\"license_redistributable\", \"first\"),\n", " unknown=(\"license_unknown\", \"first\"),\n", " url=(\"license_url\", \"first\"),\n", " )\n", " .sort_values(\"datasets\", ascending=False)\n", ")\n", "licenses" ] }, { "cell_type": "markdown", "id": "c04", "metadata": {}, "source": [ "The flags are the same for every dataset with a given license, so `first` is enough here. What the values mean:\n", "\n", "- `ncbi-geo-policy` covers data deposited in GEO. NCBI places no restrictions on the use of GEO data, but its\n", " disclaimer notes that submitters may claim patent, copyright or other rights in their data. The client counts\n", " it as allowing commercial use and redistribution, and you should check the study if your use depends on it.\n", "- `CC-BY-NC-SA-4.0` lets you share and adapt the data if you credit the authors, do not use it commercially, and\n", " share adaptations under the same license. Most HEST datasets carry it.\n", "- `CC-BY-4.0` requires credit only, and `CC0-1.0` places the data in the public domain.\n", "- `allen-institute-tou` is the Allen Institute's terms of use, which do not allow commercial use.\n", "- `unknown` means the source did not state a license that we could find. 1031 datasets, 17% of the catalog, have\n", " no known license.\n", "\n", "Read the flags of `unknown` rows with care. `license_noncommercial` is false for them, because no license\n", "forbids commercial use, but the flag does not mean that commercial use is allowed. `license_redistributable` is false\n", "because the terms are not known. Always check `license_unknown` first." ] }, { "cell_type": "code", "execution_count": 3, "id": "c05", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:05.691861Z", "iopub.status.busy": "2026-10-31T23:40:05.691450Z", "iopub.status.idle": "2026-10-31T23:40:05.702682Z", "shell.execute_reply": "2026-10-31T23:40:05.701561Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1031 datasets with an unknown license; 1031 of them have license_noncommercial == False\n" ] } ], "source": [ "unknown = df[df[\"license_unknown\"]]\n", "print(len(unknown), \"datasets with an unknown license;\", int((~unknown[\"license_noncommercial\"]).sum()), \"of them have license_noncommercial == False\")" ] }, { "cell_type": "markdown", "id": "c06", "metadata": {}, "source": [ "## Filter by license\n", "\n", "`query` applies no license filter by default, so a search does not hide datasets without telling you. Three\n", "keywords filter by license, and `license=` takes exact values like any facet:\n", "\n", "- `license_set=True` keeps the datasets with a known license.\n", "- `noncommercial=False` keeps the datasets whose known license allows commercial use. It selects the same\n", " datasets as \"Commercial use OK\" on the website. `noncommercial=True` keeps only the datasets that forbid it.\n", "- `redistributable=True` keeps the datasets whose known license allows redistribution." ] }, { "cell_type": "code", "execution_count": 4, "id": "c07", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:05.704780Z", "iopub.status.busy": "2026-10-31T23:40:05.704495Z", "iopub.status.idle": "2026-10-31T23:40:05.764102Z", "shell.execute_reply": "2026-10-31T23:40:05.763318Z" } }, "outputs": [ { "data": { "text/plain": [ "no filter 6087\n", "license_set=True 5056\n", "noncommercial=False 3869\n", "noncommercial=True 1187\n", "redistributable=True 5056\n", "license=\"CC-BY-4.0\" 333\n", "Name: datasets, dtype: int64" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "filters = {\n", " \"no filter\": {},\n", " \"license_set=True\": {\"license_set\": True},\n", " \"noncommercial=False\": {\"noncommercial\": False},\n", " \"noncommercial=True\": {\"noncommercial\": True},\n", " \"redistributable=True\": {\"redistributable\": True},\n", " 'license=\"CC-BY-4.0\"': {\"license\": \"CC-BY-4.0\"},\n", "}\n", "pd.Series({name: len(cat.query(**kwargs)) for name, kwargs in filters.items()}, name=\"datasets\")" ] }, { "cell_type": "markdown", "id": "c08", "metadata": {}, "source": [ "`noncommercial=False` leaves out the datasets with an unknown license, because their terms are not known.\n", "\n", "To check a cohort you already built, `licenses()` counts its members per license:" ] }, { "cell_type": "code", "execution_count": 5, "id": "c09", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:05.766906Z", "iopub.status.busy": "2026-10-31T23:40:05.766749Z", "iopub.status.idle": "2026-10-31T23:40:05.798239Z", "shell.execute_reply": "2026-10-31T23:40:05.797402Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
licensen
0ncbi-geo-policy116
1CC-BY-NC-SA-4.031
2unknown3
3CC-BY-4.02
\n", "
" ], "text/plain": [ " license n\n", "0 ncbi-geo-policy 116\n", "1 CC-BY-NC-SA-4.0 31\n", "2 unknown 3\n", "3 CC-BY-4.0 2" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lung = cat.query(organism=\"human\", technology=\"Visium\", tissue=\"lung\")\n", "lung.licenses()" ] }, { "cell_type": "code", "execution_count": 6, "id": "c10", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:05.800588Z", "iopub.status.busy": "2026-10-31T23:40:05.800438Z", "iopub.status.idle": "2026-10-31T23:40:05.827690Z", "shell.execute_reply": "2026-10-31T23:40:05.826898Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "152 lung sections, 118 cleared for commercial use\n" ] } ], "source": [ "lung_commercial = cat.query(organism=\"human\", technology=\"Visium\", tissue=\"lung\", noncommercial=False)\n", "print(len(lung), \"lung sections,\", len(lung_commercial), \"cleared for commercial use\")" ] }, { "cell_type": "markdown", "id": "c11", "metadata": {}, "source": [ "## Write citations\n", "\n", "Every dataset belongs to a study. `cohort.citations(path)` writes one BibTeX entry per study to a file, and\n", "`citations_text()` returns the same text with the number of entries. The entries come from `citations.bib`, a\n", "file published with each catalog release." ] }, { "cell_type": "code", "execution_count": 7, "id": "c12", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:05.830217Z", "iopub.status.busy": "2026-10-31T23:40:05.830070Z", "iopub.status.idle": "2026-10-31T23:40:06.382330Z", "shell.execute_reply": "2026-10-31T23:40:06.381504Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "65 of 152 studies have a published citation, in 8 entries after merging those that share a DOI; added 87 fallback entries that cite the dataset page; check them before publishing\n", "@article{geo:GSM7905760,\n", " title = {Distinct mural cells and fibroblasts promote pathogenic plasma cell accumulation in idiopathic pulmonary fibrosis},\n", " volume = {65},\n", " ISSN = {1399-3003},\n", " url = {http://dx.doi.org/10.1183/13993003.01114-2024},\n", " DOI = {10.1183/13993003.01114-2024},\n", " number = {6},\n", " journal = {European Respiratory Journal},\n", " publisher = {European Respiratory Society (ERS)},\n", " author = {Yang, Zhi and Cao, Gaoyuan and Tan, Xiaosheng and Orfanos, Sarah and Jude, Joseph and Barbet, Gaetan and An, Steven S. and Jiang, Dianhua and Panettieri, Reynold A. and Yang, Qi},\n", " year = {2025},\n", " month = Feb,\n", " pages = {2401114},\n", " note = {covers samples: geo:GSM7905760, geo:GSM7905761, geo:GSM7905762, geo:GSM7905763, geo:GSM7905764 and 8 more}\n", "}\n", "\n", "@article{geo:GSM8217624,\n", " title = {Early human fetal lung atlas reveals the temporal dynamics of epithelial cell plasticity},\n", " volume \n" ] } ], "source": [ "with warnings.catch_warnings(record=True) as caught:\n", " warnings.simplefilter(\"always\")\n", " bib_path = lung.citations(work / \"lung.bib\")\n", "print(*[w.message for w in caught if issubclass(w.category, UserWarning)])\n", "print(bib_path.read_text()[:900])" ] }, { "cell_type": "markdown", "id": "c13", "metadata": {}, "source": [ "The warning reports two kinds of entries.\n", "\n", "A published entry is the entry for a `study_id` in `citations.bib`. It was built from the publication's DOI and\n", "has the authors, the title and the journal.\n", "\n", "A fallback entry is written for a study that has no published entry yet, and for a dataset without a\n", "`study_id`. It cites the dataset page on the website and carries the title, DOI, year, uids and license that the\n", "catalog has. Fallback entries make sure the file cites every study in the cohort. Client versions before 0.1.2 left\n", "the studies without a published entry out, and a cohort without any published entry gave an empty `.bib` file.\n", "\n", "The bibliography has one entry per `study_id`, and for GEO data a `study_id` is a single sample, so a publication\n", "with several samples in the cohort has several entries. The client merges entries that share a DOI into one: the\n", "first entry by key keeps its key and fields, and its `note` lists the other keys it covers. The warning above\n", "therefore names more studies with a published citation than there are published entries.\n", "\n", "`citation_entries` returns the text with the two counts apart, counted after merging:" ] }, { "cell_type": "code", "execution_count": 8, "id": "c14", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:06.385157Z", "iopub.status.busy": "2026-10-31T23:40:06.384970Z", "iopub.status.idle": "2026-10-31T23:40:06.762122Z", "shell.execute_reply": "2026-10-31T23:40:06.761217Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "8 published, 87 fallback\n" ] } ], "source": [ "with warnings.catch_warnings():\n", " warnings.simplefilter(\"ignore\")\n", " text, n_published, n_fallback = citation_entries(lung)\n", "print(n_published, \"published,\", n_fallback, \"fallback\")\n", "entries = parse_bibtex(text)" ] }, { "cell_type": "markdown", "id": "c15", "metadata": {}, "source": [ "The published entry with the longest list of covered samples shows how a merged entry looks:" ] }, { "cell_type": "code", "execution_count": 9, "id": "c16", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:06.764583Z", "iopub.status.busy": "2026-10-31T23:40:06.764429Z", "iopub.status.idle": "2026-10-31T23:40:06.768457Z", "shell.execute_reply": "2026-10-31T23:40:06.767659Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "note = {covers samples: geo:GSM7905760, geo:GSM7905761, geo:GSM7905762, geo:GSM7905763, geo:GSM7905764 and 8 more}\n" ] } ], "source": [ "published = [entry for entry in entries.values() if not entry.startswith(\"@misc\")]\n", "merged = max(published, key=lambda entry: entry.count(\"geo:GSM\"))\n", "print(re.search(r\"note\\s*=\\s*\\{[^}]*\\}\", merged).group(0))" ] }, { "cell_type": "markdown", "id": "c17", "metadata": {}, "source": [ "An empty cohort raises an error instead of writing an empty file:" ] }, { "cell_type": "code", "execution_count": 10, "id": "c18", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:06.770900Z", "iopub.status.busy": "2026-10-31T23:40:06.770755Z", "iopub.status.idle": "2026-10-31T23:40:06.776282Z", "shell.execute_reply": "2026-10-31T23:40:06.775487Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ValueError: no datasets to cite: the cohort is empty\n" ] } ], "source": [ "try:\n", " lung.where(lung.members[\"n_obs\"] < 0).citations(work / \"empty.bib\")\n", "except ValueError as err:\n", " print(\"ValueError:\", err)" ] }, { "cell_type": "markdown", "id": "c19", "metadata": {}, "source": [ "## Check the entries before you publish\n", "\n", "The published entries are complete, but the fallback entries need more work. A fallback entry has a DOI when the\n", "catalog has one for the study, and then you can look up the publication and replace the entry with the\n", "publication's own BibTeX. When there is no DOI, the dataset page is the best reference we have, and the `note`\n", "field lists the uids and the license." ] }, { "cell_type": "code", "execution_count": 11, "id": "c20", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:06.778045Z", "iopub.status.busy": "2026-10-31T23:40:06.777906Z", "iopub.status.idle": "2026-10-31T23:40:06.784334Z", "shell.execute_reply": "2026-10-31T23:40:06.783574Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "@misc{10x:human-lung-cancer-11-mm-capture-area-ffpe-2-standard,\n", " title = {human-lung-cancer-11-mm-capture-area-ffpe-2-standard},\n", " howpublished = {\\url{https://spatialdata-db-web.pages.dev/datasets/visium-spatial-gene-expression-lung-pntnmjy70ylup8eq0000/}},\n", " year = {2023},\n", " note = {SpatialData-DB dataset PntNMJy70YLUP8eq0000; license CC-BY-4.0}\n", "}\n", "0 of 87 fallback entries have a DOI\n" ] }, { "data": { "text/plain": [ "geo 85\n", "10x 2\n", "Name: count, dtype: int64" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fallback = {key: entry for key, entry in entries.items() if entry.startswith(\"@misc\")}\n", "print(fallback[next(iter(fallback))])\n", "with_doi = sum(bool(re.search(r\"\\bdoi\\s*=\", entry, flags=re.IGNORECASE)) for entry in fallback.values())\n", "print(with_doi, \"of\", len(fallback), \"fallback entries have a DOI\")\n", "pd.Series([key.split(\":\")[0] for key in fallback]).value_counts()" ] }, { "cell_type": "markdown", "id": "c21", "metadata": {}, "source": [ "In this cohort none of the fallback entries has a DOI, and almost all of them are GEO samples. The `dataset_url`\n", "of a GEO dataset links its GEO series, and the series page lists the publication when the authors added one.\n", "\n", "## From the shell\n", "\n", "`sddb-cli cite` takes the same filters as `sddb-cli query` and writes the file:\n", "\n", "```bash\n", "sddb-cli cite --organism human --technology Visium --tissue lung -o lung.bib\n", "```\n", "\n", "`--catalog` selects the catalog file. The cell below runs the command with the dated file that `cat` uses." ] }, { "cell_type": "code", "execution_count": 12, "id": "c22", "metadata": { "execution": { "iopub.execute_input": "2026-10-31T23:40:06.785899Z", "iopub.status.busy": "2026-10-31T23:40:06.785762Z", "iopub.status.idle": "2026-10-31T23:40:10.325405Z", "shell.execute_reply": "2026-10-31T23:40:10.324487Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "wrote 95 citations (87 fallback; entries that share a DOI merged) -> /tmp/sddb-docs-tmp/tmpgmpq8o7a/lung-cli.bib\n", "\n" ] } ], "source": [ "command = [\"sddb-cli\", \"cite\", \"--organism\", \"human\", \"--technology\", \"Visium\", \"--tissue\", \"lung\"]\n", "command += [\"--catalog\", cat.url, \"-o\", str(work / \"lung-cli.bib\")]\n", "print(subprocess.run(command, capture_output=True, text=True, check=True).stdout)" ] }, { "cell_type": "markdown", "id": "c23", "metadata": {}, "source": [ "It prints how many entries it wrote and how many of them are fallback entries. The MCP server has the same\n", "command as its `cite` tool, see {doc}`../guides/cli-and-agents`.\n", "\n", "## Before you publish\n", "\n", "1. Run `licenses()` on the final cohort. Drop the datasets with an unknown license, or find their terms at the\n", " source (the `dataset_url` column links it).\n", "2. If any dataset is NonCommercial, make sure your use is not commercial. For `CC-BY-NC-SA-4.0`, share the\n", " data you derive under the same license.\n", "3. Write the citations, and replace fallback entries with the publication's own entry where it has a DOI.\n", "4. Cite the client from its `CITATION.cff`, which GitHub shows as \"Cite this repository\". A publication that\n", " describes the database is forthcoming.\n", "5. Record the catalog release and keep the manifest from `freeze()`, so that others can load the same datasets\n", " ({doc}`03_cohorts_and_manifests`).\n", "\n", "## Summary\n", "\n", "- `license_unknown` comes first: an unknown license has `license_noncommercial` false and is not cleared for any\n", " use.\n", "- `query` filters by license only when you ask, with `license_set`, `noncommercial`, `redistributable` or\n", " `license`.\n", "- `citations()` writes one entry per study, published where `citations.bib` has one and a fallback otherwise.\n", " Entries that share a DOI are merged. Replace fallback entries before you publish.\n", "\n", "Tutorial 4 is the last in the series. The [use cases](../use_cases/index.md) build longer analyses on the same\n", "steps, and [0. Overview](00_overview.ipynb) puts all of them on one page." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.15" }, "mystnb": { "output_stderr": "show" } }, "nbformat": 4, "nbformat_minor": 5 }