Proposal: the canonical base data store

Status: proposal 2026-10-16; adopted 2026-10-16. Nothing implements this yet. This document records the design decisions for moving base_data from its current scattered copies to one canonical, versioned store on Cloudflare R2, and is the contract the hazelbean implementation will follow. Open questions are collected at the end.

The problem

Base data currently lives in several places that disagree with each other:

the public Google Cloud bucket gtap_invest_seals_2023_04_21 (181 objects, 27.2 GiB), which get_path downloads from anonymously;
the TEEMs Google Drive Files/base_data folder, reached through the shared-data-root tier;
each person’s local ~/Files/base_data (126 GB on the maintainer’s machine);
project folders, where regenerated files accumulate.

Three properties of the current system let these copies drift apart:

get_path only checks existence. A stale local file shadows a newer remote one indefinitely, and nothing notices.
Nothing records which bytes a run used. A result cannot be traced back to the exact data that produced it.
There is no definition of “the official version”. Each copy is equally authoritative, so none is.

A fourth fact makes the cleanup unavoidable: POGs written before the overview rule of September 2026 fail the current is_path_pog level check and must be regenerated (pyramids manuscript, D.10). Most of what is in the cloud today is in that state.

Decisions at a glance

Topic	Decision
Canonical store	Cloudflare R2. Public bucket `teems-public-data` served at `https://data.teemlabs.org`; private bucket `teems-private-data` for paid access.
Initial contents	Published from the maintainer’s local `~/Files/base_data`, which is the reference copy. The Google Cloud contents are out of date and are neither reconciled nor migrated.
Source of truth	A catalogue in R2: one entry per dataset, recording its files, hashes, POG semantics, dependencies and provenance.
Intended contents	Each dataset folder in `base_data/` carries an ingestion script and the manifest it writes (`<name>_base_data_ingestion.py`, `<name>_base_data_manifest.csv`), listing that folder’s datasets with their metadata. `earth_economy_devstack/publish_base_data.py` combines every manifest, compares the result with the local files and the latest release, and publishes what is new or changed.
Writes	Only through publishing, which runs a quality gate. A lab member’s submission gets there by promotion: a curator moves it from the submissions inbox into `base_data/` and the manifest, then publishes it like any other row.
Release cadence	Every publication cuts a release. A publication may carry several datasets, so a coherent multi-dataset change is one release.
Versioning	Ref paths are stable and carry meaning, never revision numbers. The current version sits at its plain path; replaced versions are archived under `static/`, unless a publication opts out. Every run writes a lock file of what it used.
Google Drive	A generated, read-only mirror of the latest release on its own shared drive, and a submissions inbox, `base_data_submissions/`, beside `base_data/` on the NatCapTEEMs shared drive.
Submissions	A separate root, `base_data_submissions/`, never inside `base_data/`. How to submit and promote is in Contributing Base Data.
Local `base_data/`	The catalogued files, plus each dataset’s ingestion script and manifest beside them; no other code.
Subsets	`get_path(..., rung=, subset=)` reads a window by HTTP range request into a separate `base_data_cache/`.
Google Cloud	Stays up for now as a temporary fallback for refs not yet catalogued; retired later.

Vocabulary

Ref path: a path relative to the base_data root, as defined in Conventions. Code refers to data only by ref path.
Dataset: the unit that is catalogued, versioned, published and synced. One or more files that only make sense together (see Datasets).
Catalogue entry: the immutable JSON record describing one version of one dataset. The latest release says which version is current.
Content version: a dataset’s identity at the byte level, its sha256.
Release: an immutable snapshot of the whole catalogue: every dataset’s ref path mapped to its content version.
Lock file: the record, written by every run, of exactly which content version of each dataset that run used.
Manifest: a dataset folder’s CSV, <name>_base_data_manifest.csv, listing what that folder should publish, one row per dataset with its metadata. Combined, the manifests record what base data is intended to contain; R2 records what is published.
Ingestion script: <name>_base_data_ingestion.py, beside a dataset’s files, which brings them into base_data/ (converting to POGs if needed) and writes its manifest.
Publishing: the only operation that writes to R2. It turns the combined manifests and local base_data/ into a release.
Promotion: moving a submission from the submissions inbox into base_data/ under its canonical name, with a manifest row. It writes nothing to R2; publishing follows.
Curator: a person holding an R2 write token, and therefore able to publish.
Tier: public or private, meaning which bucket a dataset lives in.

The source stack

From most canonical to most local:

Layer	Location	Role	Who writes
R2 public / private	`data.teemlabs.org`, `teems-private-data`	Canonical	`hb.publish_base_data` only
Drive mirror	Shared drive EE Base Data	Generated copy of the latest release, for people who don’t program	the publish step only
Local base_data	`~/Files/base_data/`	Cache of the catalogue	`get_path`, the sync tool
Submissions	`NatCapTEEMs/Files/base_data_submissions/` on Drive, and `~/Files/base_data_submissions/`	Inbox, not yet official	lab members
Project base data	`<project>/base_data/`	Project-specific data	the project
Inputs	`<project>/input_template/`, `<project>/input/`	Tracked definitions and local overrides	the project
Outputs	`<project>/intermediate/`, `cur_dir`	Generated by tasks	ProjectFlow
Derived cache	`~/Files/base_data_cache/`	Subsets and overview extracts; disposable	`get_path`

R2 layout

The two buckets share one layout. The current version of every dataset sits at its plain ref path, so the bucket reads like base_data/. Folders the store uses for itself are reserved: static/, and anything starting with an underscore. The gate refuses ref paths that start with either.

teems-public-data/
  <ref_path>                                          the current version, overwritten by each new one
  static/<ref_path without extension>/v<release>.<ext>  archived versions, never changed
  _catalogue/<dataset_ref_path>/<release_id>.json     immutable entry, written by the release that published that version
  _release/<release_id>.json                          immutable release snapshots
  _release/latest.json                                pointer to the newest non-yanked release
  _release/yanked.json                                releases that "latest" must skip
  _staging/<ref_path>/<sha16>.<ext>                   new bytes waiting for their release to commit (normally empty)

For example, after a second release of the 1-degree area raster:

https://data.teemlabs.org/pyramids/ha_per_cell_3600sec.tif                    the current version
https://data.teemlabs.org/static/pyramids/ha_per_cell_3600sec/v2026_10_19_1.tif  the version published in release 2026.10.19.1

An archived version is named after the release that published it, with dots written as underscores. Catalogue entries are keyed by release ID too: the release ID is claimed before the entries are written, so the key is always unique, even when a dataset returns to bytes it had in an earlier version.

Archiving is optional per publication. By default, the version being replaced is copied into static/ before the plain path is overwritten. archive_previous=False skips that to save storage. The old version is then gone, and its successor’s catalogue entry records archive_key: null, so a lock file or pinned release that needs it fails clearly rather than silently.

What this layout trades. Plain paths are friendly for people, QGIS and links, and no current version is stored twice. The cost is that a plain path is overwritten in place:

A reader caught during an update can see mixed bytes. A window read is many HTTP range requests against one URL. If the plain path is replaced partway through, the reader could combine an old header with new blocks. Updates are rare and the replacement takes seconds, but readers that must be exact can guard against it: an object’s ETag changes whenever it is replaced, so a reader records the ETag at the start and sends If-Match: <etag> with each range request. R2 then refuses (412) instead of serving bytes from a newer version, and the reader starts over. (Object metadata such as the stored sha256 is not exposed through the public domain; full downloads are checked against the catalogue’s sha256 after download.)
Plain paths are cached briefly (Cache-Control: max-age=3600). Archived versions never change and are cached indefinitely.
There is a short window after each commit in which latest.json names the new version while the plain path still serves the old bytes. Publishing copies them into place straight after committing. If it is interrupted, the manifest command reports the dataset as unfinished and completes it on its next run.

Only curators hold write tokens. No bucket lock is used, so a licensing mistake, such as publishing restricted data to the public tier, remains correctable by deletion.

Datasets

Each file belongs to exactly one dataset. A file shared by many datasets, such as pyramids/ha_per_cell_<rung>sec.tif, is its own dataset, and other datasets reference it as a dependency.

Kind	Members	Dataset ref path
`pog`	One global POG, plus its observed-area companion if it has one (D.2.6)	the POG
`subpog`	One subpog	the subpog
`class_stack`	K per-class proportion POGs plus the closure class (§6)	the stack’s folder
`tileset`	Tile folder, VRT, and the coarse companion POG for rungs above the tile edge (A.6)	the VRT
`zone_set`	GeoPackage, attribute table, label POG per rung, per-zone area POGs per rung (Appendix C)	the GeoPackage
`table`	CSV or spreadsheet, plus its README	the table
`vector`	Non-canonical vector file	the file
`other`	Anything else, such as a zip of sample data	the file

A dataset moves as a unit. Publishing, download and sync are all-or-nothing per dataset. Otherwise a VRT could reference tiles that haven’t arrived, or a per-class stack could be half updated.

Tile sets keep their D.5.1 layout on R2. The VRT and its tile folder sit at their plain ref paths, so the VRT’s relative sources resolve against its own URL and /vsicurl/ on it fetches only the tiles a window touches. An archived tile set keeps the same relative layout under its static/…/v<release>/ folder. (Tile sets can’t be published yet.)

The catalogue entry

One immutable JSON file per dataset version, at _catalogue/<dataset_ref_path>/<release_id>.json. The release maps each dataset to its dataset_sha256, its version, and the release that published it, which names the entry to read:

{
  "dataset": "pyramids/ha_per_cell_10sec.tif",
  "kind": "pog",
  "tier": "public",
  "version": 2,
  "dataset_sha256": "<sha256 over the sorted member list>",
  "release": "2026.10.20.1",
  "lattice": {
    "rung": "10",
    "storage_form": "pog",
    "tile_edge_degrees": null,
    "coarse_companion": null,
    "extent": [-180, -90, 180, 90]
  },
  "semantics": {
    "variable_class": "extensive",
    "intensive_weighting": null,
    "denominator": null,
    "observed_area": null,
    "units": "ha",
    "stats": {"sum": 51007200000.0, "min": 0.0, "max": 9.56, "mean": 6.07}
  },
  "files": [
    {
      "ref_path": "pyramids/ha_per_cell_10sec.tif",
      "size": 1234567890,
      "sha256": "<64 hex>",
      "md5": "<32 hex>"
    }
  ],
  "depends_on": [],
  "conformance": {
    "pog_spec": "2026-09",
    "validator": "hazelbean <version>",
    "checks": ["is_path_pog", "is_path_pog_overview_conformant"]
  },
  "license": {"id": "CC-BY-4.0", "redistribution": true},
  "description": "Hectares of ground in each cell of the canonical 10-arcsecond global grid; the denominator of area-weighted aggregation at that rung.",
  "provenance": {
    "source": "computed from the WGS84 ellipsoid",
    "code": "hazelbean <commit> pog.write_ha_per_cell_pog",
    "promoted_from": null,
    "published_by": "jajohns",
    "published_at": "2026-10-20T14:02:11Z",
    "note": "Regenerated under the September 2026 overview rule."
  },
  "previous": {"dataset_sha256": "<…>", "version": 1, "release": "2026.10.18.1", "archive_key": "static/pyramids/ha_per_cell_10sec/v2026_10_18_1.tif"}
}

Field	Meaning
`dataset`, `kind`, `tier`	Identity. A ref path exists in exactly one tier.
`version`, `dataset_sha256`, `release`	Content version and the release that published it. `dataset_sha256` is the sha256 of the sorted `ref_path<TAB>sha256` lines of its members. Whether the dataset is current or retired is recorded in the release, not here, so that entries never change.
`lattice`	Rung, storage form (D.5.2 asks the catalogue to record it), tile edge, coarse companion, extent.
`semantics`	Variable class and weighting (D.2.4), denominator and observed-area companion as `ref_path@dataset_sha256`, units, and the exact statistics from the header (Requirement 4), so global totals need no download.
`files`	Every member file with its ref path, size, `sha256` (integrity) and `md5`. The md5 lets files be compared with Google Drive’s `md5Checksum` without downloading.
`depends_on`	Other datasets this one requires, as `ref_path@dataset_sha256`. Derived from `semantics` plus any declared extras.
`conformance`	The POG spec version and validator version it passed, and which checks ran. When the validator tightens, a sweep re-checks entries against the new spec. Not recorded yet.
`license`	Licence identifier, and whether redistribution is permitted. The gate refuses a private-tier publication without a redistribution right. Only the identifier is recorded so far.
`description`	What the data are and how to interpret them, from the manifest row.
`provenance`	Source, producing code, `promoted_from` (the submission ref path it was promoted from, or null when published straight from `base_data/`; never a curator’s local path, since the catalogue is public), who published it, when, and the release’s one-line note.
`previous`	The version this one replaced (`dataset_sha256`, `version`, `release`), or null for a first version. Following `previous` walks the whole history, so a local file can be identified as an old version rather than an unknown modification.

Versioning

Three identifiers, three jobs

Identifier	Example	Job
Ref path	`lulc/esa/lulc_esa_2020_10sec.tif`	What code uses. Human-readable and stable.
Content version	`dataset_sha256`	What the bytes are.
Release	`2026.10.20.1`	A citable snapshot of the whole catalogue.

Meaning goes in the name, revisions do not

A change in meaning gets a new ref path. A change in content only keeps the ref path and becomes a new version.

Change	Result
New source vintage (ESA 2021 alongside ESA 2020)	New ref path
New classification or recategorization	New ref path
New zone boundaries	New ref path. Correspondences are immutable once published (C.6), so `ee_r264` stays `ee_r264`.
Different rung	New ref path. The rung is in the name.
Bug fix in values	New version, same ref path
Regenerated under a newer POG spec	New version, same ref path
Recompression or metadata fix	New version, same ref path

Ref paths never carry revision tokens such as _v2, _final or _new; the gate rejects them. Identity tokens are not revision tokens: a source vintage (esa_2020), a rung (10sec) or a zone-set name (ee_r264) says what the data is.

Releases

Every publication cuts a release. Release IDs are YYYY.MM.DD.N, numbered from 1 each day.
A publication may carry several datasets. Regenerating every LULC year and its per-class stacks is one publication and one release, so no release is ever half updated.
A release is immutable. _release/<id>.json maps every current dataset to its dataset_sha256, lists retired datasets with what replaced them, and records its parent release, author, timestamp and note.
latest.json points to the newest release not listed in yanked.json.
Yanking replaces deleting. If a release turns out bad, for example because the slower data tests fail after publishing, it is added to yanked.json. “Latest” then skips it, and the fix is a new publication.
Release notes are generated from each publication’s required note and appended to Base Data release notes.
Retiring replaces deleting. Retiring a dataset is a publication whose release moves it from the current map to the retired map, with what replaced it. Its entries remain under _catalogue/, and its last version is archived under static/ unless the retiring publication says otherwise.

What a user experiences

A user who knows nothing about versioning sets nothing. At the start of a run, get_path fetches _release/latest.json once and compares each local base_data file it resolves against that release:

current → used;
older version, or missing → the latest version is downloaded, and the update is logged (“updated lulc/… to 2026.10.20.1”);
offline → the local file is used, with a warning.

p.base_data_policy changes this: 'update' (the default, as above), 'warn' (report stale files but don’t download), or 'offline' (never contact R2).

Every run writes a lock file, <project_dir>/base_data_lock/<run timestamp>.json, listing each dataset resolved from a base-data tier with its dataset_sha256, the release it came from, and its source (catalogue, legacy_gcs, submissions, or local_unpublished). This is automatic and invisible. It gives reproducibility after the fact: reproducing a result years later means pointing the project at the lock file from the original run (p.base_data_lock_path = ...), and get_path then fetches those exact bytes: from the plain path if that version is still current, otherwise from its static/ archive. A version replaced with archive_previous=False can’t be fetched again, and get_path says so.

Pinning is for deliberately frozen work, such as a paper in revision, a team that must all see the same data, or test suites:

p.base_data_release = '2026.10.20.1'

The release ID is also what a paper cites (“EE base data release 2026.10.20.1”).

Publishing

Manifests and ingestion scripts

Each dataset folder in base_data/ carries two files beside its data, and optionally its raw input:

base_data/lulc/esa/
    lulc_esa_base_data_ingestion.py    raw input -> the files below, then writes the manifest
    lulc_esa_base_data_manifest.csv    one row per dataset in this folder
    lulc_esa_1992.tif ... lulc_esa_2023.tif
    raw/                               the raw input, exactly as downloaded (never uploaded)

raw/ holds the input exactly as provided at its download link, placed there by hand; the ingestion script reads it, records where each file comes from, and writes elsewhere. Publishing, the manifests and the catalogue ignore it, and nothing in it is ever uploaded. The rules are in Conventions, under “Base data dataset folders and raw/”.

There are three kinds of ingestion script:

A raw-data ingest that lives entirely in base_data/, such as the ESA land-cover maps: convert raw/ to POGs (skipping years already converted), then list. A step whose history was never recorded, like the ESA maps before 2016, is a function that states that gap rather than being left out.
A thin call to code in another repo, such as the pyramids’ pyramids_base_data_ingestion.py, which writes any missing area raster with hazelbean’s hb.write_ha_per_cell_pog and lists the rasters. Larger production code stays in the repo that owns it.
A bespoke script that moves outputs of a finished, published project into base_data/ and lists them (in time perhaps a simple rule, such as “everything in the project’s output directory”).

Most scripts end with one call, hb.add_directory_to_base_data(...). It places or keeps the files under their ref paths, converts rasters that aren’t POGs yet (in place, deleting each original once its POG is written), or leaves that to publishing with make_pogs=False, and adds a row per new dataset to the folder’s manifest. It skips code, manifests, GDAL sidecars, raw/ and leftover displaced or temporary files, and can be limited by include patterns or kept out of subdirectories. It never writes to R2, and ends with the publishing dry run.

A manifest is UTF-8 with a BOM, like every EE Spec CSV:

Column	Meaning
`ref_path`	The dataset’s full ref path. The local copy is read from `~/Files/base_data/<ref_path>`.
`kind`, `tier`	Dataset kind (see Datasets) and bucket tier. Only `pog` and `public` can be published so far.
`variable_class`	For rasters: `extensive`, `intensive`, `categorical` or `covariate`. Publishing uses it to convert a raster that isn’t a POG yet.
`license`, `units`, `data_source`, `description`	Required metadata, copied into the catalogue entry. A licence that isn’t a standard identifier is given by name and URL.
`code`	The script that produced the files.

Curators publish with earth_economy_devstack/publish_base_data.py. It is about all datasets rather than any one, so it lives in the devstack, and its one call is:

hb.publish_base_data_from_manifests(note="...", dry_run=True)

It finds every *_base_data_manifest.csv under base_data/ and combines them; a ref path listed by two manifests is an error. On a real run it first converts any listed raster that isn’t a POG yet, using its row’s variable_class (a dry run reports it as needs_pog). It then reports every row as new, changed (local bytes differ from the published version), unchanged, unfinished, missing_locally or unsupported, and every published dataset that no manifest lists as not_in_manifest, and publishes the new and changed rows together as one release, each with its own row’s metadata. dry_run defaults to True, so the report is the plan and publishing is always an explicit second run. Editing only a row’s metadata, with the bytes unchanged, does not yet publish a new entry.

The function

hb.publish_base_data(
    sources,          # local POG paths, one per dataset
    ref_paths,        # destination dataset ref paths, same order
    note,             # required one-line change note
    license, units, data_source, description, code=None,   # each given once, or once per dataset
    dry_run=False,    # run the gate and report, write nothing
)

publish_base_data is the only code that writes to R2, and the manifest command is a thin loop over it. Data reaches it by two routes:

Local route. A curator produces or regenerates files in their own base_data/, adds or updates the manifest rows, and publishes. This is the common case, and the only one built so far.
Drive route. A curator first promotes a lab member’s submission: reviews it, makes it conform, moves it into base_data/ under its canonical name, and adds its manifest row with promoted_from set to the submission’s ref path. Publishing then proceeds as for the local route. The function for this, hb.promote_base_data_submission, is reserved for step F and not built.

Steps

Resolve the members of each dataset and its declared dependencies.
Gate: every check in the table below. Any failure stops the whole publication, and nothing is written.
Hash every member file, computing sha256 and md5.
Stage each new file under _staging/. A staged copy that already exists is reused. Upload is multipart for large files.
Archive each version being replaced, by a server-side copy from its plain path to static/…/v<release>.<ext>, unless archive_previous=False. The gate has already checked that the plain path holds the published version.
Claim the release ID and commit by the protocol in Concurrent publications, writing the catalogue entries in between.
Move into place: copy each staged file over its plain path (server-side), then delete the staged copy. Nothing at a plain path changes before the commit.
Rebuild the pages of https://data.teemlabs.org (see Browsable pages), when the run is given site_dir.
Publish the changed datasets to the Drive mirror (not built yet).
Append the release notes (not built yet).

Concurrent publications

Two curators may publish at the same moment. R2 evaluates conditional PutObject headers atomically against the bucket’s committed state, so If-None-Match: * (“only if this key does not exist”) and If-Match: <etag> (“only if this object is unchanged”) make one compare-and-swap commit possible with no lock object:

Read _release/latest.json and keep its ETag. It names the parent release.
Build the new release snapshot from the parent plus this publication’s changes, with the next release ID: the parent’s plus one if it was cut today, else .1.
Write _release/<id>.json with If-None-Match: *. This claims the ID; a 412 Precondition Failed means another publication took it.
Write each catalogue entry to _catalogue/<ref_path>/<id>.json with If-None-Match: *. Only this publication holds the ID, so these keys are free.
Write _release/latest.json with If-Match: <etag from step 1>. A 412 means another publication committed since step 1, so this release was built on a stale parent and would silently drop the other publication’s changes.

On a 412 at any step, the release file and entries this publication wrote are deleted (nothing references them yet) and it stops with “another release was published first; rerun”. Rerunning rebuilds on the new parent and skips files already uploaded. With few curators this is rare, so stopping is preferred over retrying automatically.

Step 5 is the only commit point. The same If-Match rule applies to every rewrite of yanked.json. A lost commit leaves the plain paths untouched: only staged files and archive copies remain, both harmless and reused by the rerun.

The gate

Check	Applies to	Implemented by
Conforms to the POG spec, full check: variable class declared, nodata convention, overview levels and their numerical rule, exact statistics, COG layout	pog, subpog, class_stack, tileset members	`hb.is_path_pog(full_check=True)`, `hb.is_path_subpog`, `hb.is_path_pog_tileset`
Name is lowercase and relative; no revision tokens; does not start with `_` or `static/`	all	gate
Ref path unique across both tiers	all	catalogue lookup (not yet: only the public tier exists)
Every dependency already catalogued, or included in the same publication	all	release lookup
Zone set canonical at its stated rung (C.7)	zone_set	gate
Licence, units, data source and description given, from the manifest row	all	gate
Redistribution permitted for the private tier	private tier	gate (not yet)
Not the same bytes as the current version (a no-op publication is refused)	all	hash comparison

The first dataset published must be the pyramids/ha_per_cell_<rung>sec.tif stack. Every area-weighted intensive POG names it as its denominator, so no intensive raster can pass the dependency check before it is catalogued.

Browsable pages

https://data.teemlabs.org serves human-readable pages beside the data, built by hb.write_base_data_index_pages from the manifests and the latest release, and rebuilt by every real run of publish_base_data.py:

index.html, the landing page: a custom HTML page (docs/base_data/_landing.html) describing the collections (the top-level folders, with titles and text from docs/base_data/_collections.yml), the store, and how to read from it.
<folder>/index.html for every folder holding published datasets or above one: docs/base_data/_folder_page.qmd filled in and rendered by Quarto into one self-contained file. It shows the folder’s text, cards for its subfolders, its ingestion script’s docstring as an About section, and a table of its published manifest rows with each file’s version, release, size, resolution and archived versions. Metadata shared by every row is shown once.

The pages are cached for five minutes. R2 doesn’t serve index.html for a folder path, so a Cloudflare URL Rewrite rule on the zone appends index.html to any path ending in /. How to edit the pages is in Contributing Base Data.

Google Drive

Where the mirror and the inbox live

Permissions in a shared drive pass down to every folder in it. The mirror is generated and must not be edited, so it gets a shared drive of its own. The inbox stays on the existing NatCapTEEMs shared drive, as Files/base_data_submissions/ beside Files/base_data/ (decided 2026-10-20), so lab members need no new drive; that they never write into base_data/ there is a convention rather than a permission:

Location	Lab members	Curators and publish step
EE Base Data shared drive (mirror)	Viewer	Content manager
`NatCapTEEMs/Files/base_data_submissions/`	Contributor	Content manager

The old NatCapTEEMs/Files/base_data/submissions/ folder is retired; its contents move into base_data_submissions/.

The mirror

It contains the latest release, laid out by ref path exactly as base_data/ is. Storage is not a constraint (50 TB available), so it mirrors everything, including 1-arcsecond tile sets.
It is generated, never edited. The publish step after each release updates the changed datasets and removes retired ones.
It includes a human-readable index (index.csv and an HTML page) listing every dataset with its units, licence, release and description, for people who don’t program.
It can be configured as a shared data root (HB_SHARED_DATA_DIRS). Because every file has a catalogued hash, get_path verifies a mirror copy before trusting it. hb-setup-machine-env is updated to find the new mirror instead of the old NatCapTEEMs/Files/base_data folder, which the mirror replaces.

Submissions

Lab members never write into base_data/. They put data in base_data_submissions/<project>/ on the NatCapTEEMs shared drive, keeping a local copy in ~/Files/base_data_submissions/<project>/, together with a README.

Their code refers to it by a ref path that starts with base_data_submissions/:

p.emissions_factors_ref_path = os.path.join('base_data_submissions', 'air_quality', 'emissions_factors.csv')

get_path resolves base_data_submissions/… refs from the local submissions folder, then the mounted submissions drive. It never resolves them from R2.
Redirect after promotion. Promotion usually changes the ref path, because the rung suffix is added and the name standardised. The catalogue’s provenance.promoted_from records the submission path. When a base_data_submissions/… ref no longer resolves, get_path looks it up there, logs “promoted to <new ref path>; update your code”, and resolves to the promoted dataset. Code keeps working on the day of promotion.
Contributing Base Data is rewritten to this design (2026-10-20). Until get_path searches the inbox, contributors resolve submission refs with prepend_possible_dirs=[~/Files], as that page shows, and curators promote by hand.

The invariant this buys: local ~/Files/base_data/ holds catalogued files plus each dataset’s ingestion script and manifest. Any other file found there is unambiguously an orphan or a local modification, and the sync tool can say which.

get_path resolution order

Searched first to last:

#	Root	Role	Checks
1	`cur_dir`	The task’s own output	none
2	`intermediate_dir`	Earlier tasks’ outputs	none
3	`input_dir`	Untracked project overrides	existing stale-template warning
4	`input_template_dir`	Tracked project definitions	none
5	`project_base_data_dir`	Project-specific data	none. New: defined today but not searched.
6	`caller_dir`, `cwd`	Regenerated files about to replace base data	warns when it shadows a catalogued ref path with different bytes
7	submissions roots	Only for `base_data_submissions/…` refs; redirect if promoted	none
8	`base_data_dir`	Local cache of the catalogue	freshness against the release, per `base_data_policy`
9	shared data roots (Drive mirror)	Copy into `base_data_dir`	hash verified
10	R2 public	Download, or window read when `rung` or `subset` is given	hash verified on download
11	R2 private	As 10, only if credentials are configured	hash verified on download
12	Google Cloud (temporary)	Only for refs not in the catalogue	logs “legacy source”; lock file records `legacy_gcs`

Notes:

Shadow warnings, not prohibitions. A hit at tier 6 for a ref path the catalogue knows, with different bytes, is a legitimate workflow: someone has regenerated a file that will be published. It is logged once per run, and the lock file records it as local_unpublished with its hash. The run is then visibly marked as having used unpublished data, and nothing is silent.
Hashing is cached. A local index outside base_data/ (under ~/.cache/hazelbean/) stores (path, size, mtime) → sha256, so multi-gigabyte files are hashed only when they change.
The release is fetched once per run, and served from an ETag-validated local copy when unchanged.
Tier 12 is removed in the hazelbean release that accompanies retiring Google Cloud.

Subset and overview reads

The call

p.get_path('lulc/esa/lulc_esa_2020_10sec.tif', rung=900, subset=(-130, 40, -120, 50))

rung= reads the stored overview for that rung (identified by its dimensions, A.1) instead of the base level. Over HTTP that is a handful of range requests rather than a full download. It must be a rung in the file’s overview chain.
subset= restricts the read to a window:

Form	Example	Resolved as
Bounding box	`(xmin, ymin, xmax, ymax)`	Snapped outward to whole degrees; returns a subpog
Tile corner string	`'40N_130W_10_10'`	Exact extent
Zone	`('cartographic/ee/ee_r264.gpkg', [840, 124])`	Bounding box from the zone set’s per-zone bbox column, plus the per-zone-area subpog as a companion
Polygon	a shapely geometry	Bounding-box subpog, plus an overlap-fraction companion

Rules

Snap to whole degrees, not to the rung. D.5.1 names subpogs by whole-degree corner and size, so the cache key is always expressible. Snapping coarser also means nearby requests share files. At rungs of 18000 s (5°) and coarser the global file is at most 72 × 36 cells, so the whole file is fetched.
Never mask values. A polygon or zone subset returns the full bounding-box data plus a weights companion. Clipping by centroid or all-touched would reintroduce the zonal error that §4.2.3 removes. Under late binding (§4.5.1), the weights are applied at the end of the computation.
Overviews are copied, not recomputed. A subpog’s overview chain is the global chain truncated to levels whose cell divides its extent (hb.get_pyramid_overview_levels_for_bb). Because the extent is snapped to whole degrees, each of those levels is an exact window of the source’s own overview, so the windows are read and written with OVERVIEWS=FORCE_USE_EXISTING.
Sparse data reads as zero. Tiles missing from a tile set’s VRT read as 0, or as the covariate nodata value for covariate rasters (A.6). A window entirely inside missing tiles returns a valid all-zero subpog, not an error.
Rungs coarser than a tile set’s tile edge are read from the dataset’s coarse_companion (A.6).
Size guard. max_subset_cells, with a default to be set and overridable per call, stops accidental requests such as reading a 1-arcsecond continent. It is a safety rail, not access control: anyone with read access can bypass hazelbean.
This closes D.10(c), “range-request reads of a window from a remote global POG through hb.get_path”.

Mechanics

Public tier: /vsicurl/https://data.teemlabs.org/<ref_path> for the current version, or its static/…/v<release>.<ext> archive for a pinned older one. Window reads send If-Match with the ETag they started with, so a publication that replaces the file mid-read makes them restart rather than mix versions; full downloads are checked against the catalogue’s sha256.
Tile sets: /vsicurl/ on the VRT at its plain path.
Private tier: /vsis3/teems-private-data/<ref_path> with the R2 endpoint and per-person credentials (see Tiers).
GDAL configuration, set by hazelbean around remote reads:

Option	Value	Why
`GDAL_DISABLE_READDIR_ON_OPEN`	`EMPTY_DIR`	Stops GDAL from trying to list the remote directory, which fails on R2
`CPL_VSIL_CURL_ALLOWED_EXTENSIONS`	`.tif,.vrt`	Avoids probing for sidecar files
`GDAL_HTTP_MULTIRANGE`	`YES`	Fetches several blocks per request
`GDAL_HTTP_MERGE_CONSECUTIVE_RANGES`	`YES`	Merges adjacent block reads
`VSI_CACHE`	`TRUE`	Caches fetched ranges within a process

base_data_cache

All subset and overview reads are written to a user-wide, disposable folder, ~/Files/base_data_cache/, never to base_data/. The folder can be overridden with HB_BASE_DATA_CACHE_DIR in machine.env.

Layout

One folder per source dataset, then one file per rung and extent:

base_data_cache/lulc/esa/lulc_esa_2020_10sec/
    900sec_40N_130W_10_10.tif
    900sec_20N_30E_30_60.tif
    10sec_40N_130W_10_10.tif
    10sec_40N_130W_10_10_zone_ee_r264_840.tif     per-zone-area companion
    10sec_40N_130W_10_10_geom_7c1e09ab.tif        overlap fraction for an arbitrary polygon (geometry hash)

Each file records its origin in its metadata: HB_SOURCE_REF_PATH, HB_SOURCE_SHA256, HB_SUBSET and HB_RUNG.

Behaviour

The catalogue gives the source’s current dataset_sha256.
If the cache file exists and its HB_SOURCE_SHA256 matches, it is returned with no network traffic.
Otherwise the window is read from the best available source (a current local copy in base_data/, else R2), written to a temporary name, then renamed into place. Parallel tasks requesting the same subset therefore never see a half-written file.

Rules

Never synced, published, mirrored or catalogued. Deleting the whole folder is always safe; it refills on demand.
Invalidation is lazy. When a source’s hash changes, the next request sees the mismatch and fetches again. The sync tool can also sweep stale entries, and one folder per dataset makes that a single folder delete.
Optional size cap. HB_CACHE_MAX_GB evicts least-recently-used files.
Overlapping subsets are stored separately. That duplication is accepted. A later optimisation can serve a request contained in an existing cached subset at the same rung by cutting it locally with hb.write_subpog_from_pog_window.
Why not base_data/: a subset at a canonical ref path would look like a corrupted or modified copy of the full file, and would shadow it.
Why not the project’s intermediate/: many projects ask for the same windows, so sharing the cache across projects avoids repeat downloads.

Tiers and access

	Public	Private
Bucket	`teems-public-data`	`teems-private-data`
Address	`https://data.teemlabs.org`	R2 S3 endpoint; later a Worker at a separate subdomain
Read access	Anyone	Per-person R2 token now; Worker-issued presigned URLs when access is sold
GDAL path	`/vsicurl/`	`/vsis3/`, or `/vsicurl/` with a presigned URL
Catalogue	`_catalogue/` in its bucket	`_catalogue/` in its bucket

GDAL has no permission system of its own. It signs requests with whatever credentials it finds, and R2 decides. For the private tier, machine.env (never committed) holds AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_S3_ENDPOINT=<account>.r2.cloudflarestorage.com and AWS_VIRTUAL_HOSTING=FALSE. Revoking a person’s token in the Cloudflare dashboard makes their next request fail with 403.
Enforced limits require a server. When access is sold, a Cloudflare Worker in front of the private bucket checks a subscriber key, issues short-lived presigned URLs, and enforces a byte quota per key per day. Limiting by extent is only practical through tiles: the private tier can serve tile sets only, and quota whole tiles.
The public tier needs no limit. Egress is free and requests are cheap.
A ref path exists in exactly one tier, so code never names a tier. get_path tries public, then private if credentials exist.

The sync and status tool

A command that compares local base_data/, the catalogue (latest or a pinned release), the Drive mirror and the submissions drive, and labels every dataset:

Label	Meaning	Offered action
`current`	Local matches the release	none
`stale`	Local hash matches an older version (found by following `previous`)	`pull`
`missing`	In the release, not local	`pull` (optional)
`modified`	Local hash matches no known version	publish, or revert
`orphan`	In `base_data/`, in no manifest, and not an ingestion script or manifest	move to submissions, or list it with an ingestion script and publish
`mirror_stale`	Drive mirror lags the release	`republish-mirror`
`submission_pending`	Waiting in the submissions drive	review, promote, then publish

Nothing ever moves upward automatically. pull and republish-mirror can run unattended; publishing is always an explicit, non-dry run of the manifest command, which already reports the upward half of these labels (new, changed, unchanged, missing_locally, not_in_manifest).

The first version is read-only and useful before R2 holds anything: it hashes local base_data/ through the cached index and compares it with the Drive md5s (from rclone lsjson --hash), showing where the old Drive folder differs from the reference local copy.

Google Cloud transition

Now: the bucket stays up and unchanged. Its contents are out of date relative to the maintainer’s local copy, so they are not inventoried, reconciled or migrated.
During transition: get_path tier 12 falls back to Google Cloud for refs not yet catalogued, with a warning, and the lock file records the source as legacy_gcs.
At retirement, on an announced date:
1. Snapshot the bucket to an R2 bucket ee-legacy-gcs with Cloudflare Super Slurper. This is free apart from R2 Class A operations and Google’s egress, which is a few dollars at 27 GiB.
2. Add an indefinite bucket lock and keep ee-legacy-gcs private. get_path never reads it.
3. Release the hazelbean version without tier 12.
4. Delete the Google bucket. Older hazelbean installs, which hardcode it, stop finding data from that date.

Implementation order

Step	Work	Where
A	Add `teemlabs.org` to Cloudflare; create `teems-public-data`; connect `data.teemlabs.org`; create a curator token	Cloudflare and hosting.com accounts
B	This document, reviewed and adopted	`earth_economy_devstack/docs`
C1	Read-only status tool: cached local hashing, compared with Drive md5s	hazelbean
C2	`publish_base_data`, folder manifests with `add_directory_to_base_data` and `publish_base_data_from_manifests`, local route, `pog` datasets. First publication: the `ha_per_cell` stack. Built in `hazelbean/base_data_processing.py`.	hazelbean, devstack
D	`get_path` R2 tier: release fetch, download with hash check, freshness policy, lock files, `project_base_data_dir` search, shadow warnings, temporary Google Cloud tier	hazelbean `project_flow.py`, `cloud_utils.py`
E	`rung=` and `subset=` window reads, `base_data_cache`, size guard; then `tileset`, `class_stack` and `zone_set` datasets	hazelbean
C3	Browsable pages at `data.teemlabs.org`: `hb.write_base_data_index_pages`, templates in `docs/base_data/`, rebuilt after every real publish. Done.	hazelbean, devstack
F	Drive mirror publishing, `get_path` submissions root and redirect, `hb.promote_base_data_submission`, and the `get_path` section of the conventions (`contributing_base_data.qmd` is already rewritten)	hazelbean, docs
G	Private tier: per-person tokens, then a Worker with byte quotas	Cloudflare, hazelbean

Open questions

Default max_subset_cells.
Lock-file location and retention. <project_dir>/base_data_lock/ with one file per run is proposed; whether old lock files are pruned is open.
Who the curators are, meaning who holds R2 write tokens.
Catalogue scale. One JSON file per dataset version, with full snapshots per release, is simple and adequate for thousands of datasets. If releases become large, snapshots can store only changes relative to their parent.
CORS on the public bucket, needed only if browser-based maps will read from it.
Private bucket name. teems-private-data is proposed to match teems-public-data; it is not created yet.
Whether to keep the Google Cloud fallback (tier 12). It keeps refs working for users without a local copy until R2 is populated, but it serves data known to be out of date. The alternative is to drop it now and let uncatalogued refs fail with a clear message.