Base Data

The Earth-Economy Devstack maintains a single shared, curated data store, base_data, that all projects in the stack draw from. Rather than each project bundling (or losing track of) its own copies of common inputs, code references data by a ref path (a path relative to the base_data root) and ProjectFlow’s get_path() resolves it: locally if present, from a shared data root if one is mounted, and otherwise by downloading it from our cloud storage automatically. The full resolution order is defined in Conventions.

The published store, with every dataset, its metadata and its download link, is at https://data.teemlabs.org.

Two properties make base_data more than a pile of files:

  1. It is automatically downloadable. Any project that references data by ref path “just works” on a fresh machine: clone the repo, run it, and get_path() fetches what is missing. This is the backbone of our replicability story.
  2. All rasters are Pyramidal Cloud-Optimized GeoTIFFs (POGs). Every raster spans the globe on one of a fixed set of nested resolutions with bit-identical geotransforms, which enables downloading a subset of a single raster, exact and very fast zonal statistics, and accurate area-per-cell calculations without ever using projected data.

What’s in it

Data are grouped into collections, one per top-level folder. The published ones are listed, with their datasets, at https://data.teemlabs.org:

  • pyramids: the canonical ha_per_cell_<rung>sec.tif grids the POG system is built on.
  • lulc: land-use and land-cover time series, starting with the ESA CCI / C3S annual maps.

More of the local store (cartographic, gtappy, luh2, iams, crops, soil, seals, global_invest, gtap_invest, invest_sample_data) is published as each folder gets its ingestion script. Within these, files follow the EE Spec naming and organization rules in Conventions, so a ref path is predictable from what the data is.

How data get in

Each dataset folder carries its own lineage: an ingestion script (<name>_base_data_ingestion.py) that turns the raw input in its raw/ folder into POGs, and the manifest it writes (<name>_base_data_manifest.csv) listing each dataset with its metadata. publish_base_data.py in this repo combines every manifest, publishes what is new or changed as one release, and rebuilds the pages at data.teemlabs.org. The rules are in Conventions, under “Base data dataset folders and raw/”, and the design in the base data store proposal.

The pages there are built from this folder: _landing.html (the landing page), _folder_page.qmd (every collection and dataset folder page), _styles.css, _collections.yml (the collections’ titles and descriptions) and teems_logo.png. The leading underscores keep this site from rendering them itself. How to edit them is in Contributing Base Data.

Contributing

Lab members can contribute their project’s input data so their work runs seamlessly for everyone else, by putting it in the base_data_submissions/ inbox on the NatCapTEEMs shared drive: see Contributing Base Data for the submissions workflow and for how curators promote and publish, and POGs for the raster format contributions are cleaned into.

Changes to the store itself are tracked in the Base Data release notes.