Proposed changes

Status: design specification. No code implements this yet. The canonical parser, run-step iterator, model-spec registry and iterator builder will live in hazelbean (next to project_flow.py); seals, gtappy, gtap_invest and global_invest will import them and delete their local copies of assign_df_row_to_object_attributes.

This document holds the two companion design specs for scenario-driven runs, as layers:

  1. Scenario definitions v2 — the CSV instance layer: the schema researchers edit, one row per scenario, with the canonical clock and model-clock harmonization.
  2. Model specs and iterator-driven staging — the model-schema and execution layers above it: each model’s typed inputs, outputs, clock and iteration role, and how the task tree is built from them.

Scenario definitions v2 — the CSV instance layer

This spec defines version 2 of the scenario definitions CSV that drives SEALS and the models built around it (gtap_invest, global_invest, gtappy, ngfs_pnas). It replaces the v1 standard_scenarios.csv schema and its per-project variants.

Example files live in examples/scenario_definitions_examples/:

  • single_stage_seals.csv — a SEALS-only project (migrated from the v1 SEALS template), including a hindcast year.
  • multi_stage_ngfs.csv — a GTAP+SEALS project (migrated from the NGFS/PNAS variant): a 2020-anchored canonical clock with a gtap stage clock (2023 base, annual via gtap_timestep=1).
  • hindcast_example.csv — minimal demonstration of backward years.

Why v2

The v1 schema accumulated five structural problems as projects grew:

  1. Per-model time columns multiplied ad hoc. The NGFS variant needed key_base_year/years for the CGE model (2023→2050) and seals_key_base_year/seals_years for SEALS (2020→2050). Every new model stage added more one-off columns.
  2. “baseline” conflated two different things: the observed historical anchor (base-year LULC data, calibration inputs) and the business-as-usual counterfactual trajectory. One is data; the other is a scenario.
  3. scenario_type was overloaded with workflow control. Values like baseline_ignore_dependencies smuggled a label and a dependency flag into the type field, and pseudo-baseline rows were wired together through comparison_counterfactual_labels.
  4. Years only ran forward from the base year; hindcasting (simulating years before the anchor) was impossible to express.
  5. Model clocks had no defined meeting point. GTAP’s database anchors at 2023 and solves annually; SEALS anchors on observed LULC (ESACCI ends at 2022) and allocates in 5–10-year steps; MAgPIE delivers 5-year steps anchored at 2020. Nothing specified how these timelines exchange data, so harmonization lived in ad hoc task code: the seals_years override in seals_utils.py:598-607, three duplicated linear-interpolation blocks in gtappy_tasks.py, and “POSSIBLE HACK” year-filter fallbacks in seals_process_coarse_timeseries.py.

v2 keeps a single flat, Excel-editable CSV — duplication across rows is accepted — and fixes the semantics.

Core concepts

A v2 file is a table where every row is either an observed anchor or a runnable scenario:

  • An observed row defines the historical anchor: the base year, the base-year LULC map, calibration inputs, and any additional observed years. It runs no projection years. It is data preparation, not a counterfactual.
  • A bau or policy row is a trajectory to simulate. It points at its anchor (observed_reference_label), optionally at scenarios that must run first (depends_on), and at scenarios it is compared against in reporting (compared_to).

Real computations that were previously disguised as baseline rows (e.g. an initial CGE equilibrium run) become ordinary bau rows that other rows depends_on. Observed anchors get observed rows; real runs get bau/policy rows.

Every scenario also defines exactly one canonical clock — one base_year and one years list that the whole project exchanges data on. Model stages may run on their own native clocks (GTAP annually from 2023, SEALS decadally from an observed-LULC anchor), but anything that crosses a model boundary is expressed on the canonical clock, with declared interpolation for the gaps. See The canonical clock and model-clock harmonization.

Column reference

Identity

column type required semantics
scenario_label label yes Primary key, unique within the file. Rows whose label starts with # are comments and are skipped entirely.
scenario_type enum yes Closed vocabulary: observed, bau, or policy. Anything else is a load-time error.
tags label list no Free-form grouping (e.g. stress_test, ngfs_phase4) for filtering and reporting. Never read by run logic.

Comparison and dependency DAG

column type required semantics
observed_reference_label label on bau/policy rows Which observed row anchors this scenario (base-year data and calibration lookup). Replaces v1 baseline_reference_label.
depends_on label list no Scenarios that must complete (all stages) before this row runs. Defines the execution DAG; the parser topologically sorts, and cycles are a load-time error.
compared_to label list required on policy rows Scenario(s) this row is differenced against in reporting/comparison tasks. Replaces v1 comparison_counterfactual_labels.

These are three separate concerns that v1 mixed together: where is my anchor data, what must run before me, and what am I compared against.

Time

Time is specified at two levels: the canonical clock (the plain columns) and optional stage clocks (stage-prefixed columns) for models whose data availability or cadence differs. The canonical clock is the project’s single source of truth — cross-stage data exchange, comparison tasks and reporting all speak canonical years. Stage time columns are declarations of native data availability, and the framework — not task code — owns the mapping between each stage clock and the canonical clock (see the harmonization section).

column type required semantics
base_year int yes The single canonical anchor year. Must be a year the anchoring observed row can supply data for — either truly observed (with ESACCI LULC that means ≤ 2022) or constructed by an upstream row (see Constructed anchors). Replaces v1 key_base_year. Must never appear in years.
observed_years int list no Additional observed/historical years available for calibration or validation. Replaces v1 base_years. Defaults to just base_year.
years int list on bau/policy rows Canonical years to simulate and report. May contain years before and/or after base_year (see Hindcasting). Empty on observed rows, and may be empty on a bau row that is a base-year-preparation run only (e.g. an initial CGE equilibrium).
<stage>_base_year int no The stage’s native anchor (e.g. gtap_base_year=2023, the GTAP database year). Empty → canonical base_year.
<stage>_years int list no The stage’s native run years. Empty → canonical years. Must never contain the stage’s resolved base year. Mutually exclusive with <stage>_timestep.
<stage>_timestep int no Excel-friendly alternative to <stage>_years: expands to every <stage>_timestep-th year after the stage base year, through max(years) (e.g. gtap_timestep=1 with gtap_base_year=2023 → 2024 2025 … 2050).
<stage>_interpolation enum no How the stage’s published outputs are filled at canonical years its native clock skips: linear (default), nearest, hold (carry the previous native value), compound (for growth-rate variables: aggregate by compounding, split by geometric root — never linearly; see Exogenous shocks), none (a skipped canonical year is a load-time error).
<stage>_extrapolation enum no How canonical years outside the stage’s native range are filled: hold (default), linear, error.
coarse_interpolation enum no Same vocabulary as <stage>_interpolation, applied when extracting coarse-projection NetCDF bands (MAgPIE, LUH2) at years the source lacks. Default linear. Replaces the hardcoded decadal interpolation in the BtC extraction path and the silent year-skipping in extract_global_netcdf.

Derived at load, never written in the CSV: previous_years — for the canonical clock and for each stage clock, a dict mapping each year to the adjacent year toward the base year in its chain (see Hindcasting and the ProjectFlow attribute contract).

Per-stage overrides

When a project chains multiple model stages with different timesteps (e.g. GTAP runs 2023→2050 while SEALS runs 2020→2050), any time or config field can be overridden per stage by prefixing the stage name:

<stage>_<field>     e.g.  gtap_base_year, gtap_years, gtap_model_label

Because single underscores are ambiguous (regional_projections_input_path is not a “regional” stage override), the parser requires a registered stage list, declared in the run file before the CSV is loaded:

p.scenario_stages = ['gtap', 'seals']

Resolution rule when executing stage s, field f: use the <s>_<f> cell if it is non-empty, otherwise fall back to the plain <f> column. The plain columns are the single-stage default — a SEALS-only project never writes a prefix. The type of a prefixed column is inherited from its base field name (gtap_shocks_path is a path because shocks_path would be).

For time fields the fallback rule is the same (empty stage cell → canonical column), but the semantics are stronger than a value swap: stage time columns declare a native clock that the framework validates against and resamples to the canonical clock (see the harmonization section). For every other config field the override is a simple per-stage substitution.

Naming and provenance labels

Unchanged from v1; these compose output directories as exogenous_label/climate_label/model_label/counterfactual_label/year:

exogenous_label, climate_label, model_label (the coarse-projection source model, e.g. luh2-message, magpie), counterfactual_label (the trajectory name, e.g. bau, below_2c).

Spatial extent

aoi (ISO3 code, global, or a vector path), regions_vector_path, regions_column_label.

Coarse projection sources

Unchanged names from v1: coarse_projections_input_path, coarse_src_label, coarse_simplification_label, coarse_correspondence_path, lc_class_varname, dimensions, time_dim_adjustment (e.g. add2015, arithmetic applied to NetCDF time coordinates).

Base-year LULC and calibration

Primarily meaningful on observed rows: base_year_lulc_path, lulc_src_label, lulc_simplification_label, lulc_correspondence_path, calibration_parameters_path (renamed from v1 calibration_parameters_source so the _path suffix types it correctly).

Regional / vector stage

regional_projections_input_path (templatable, see Templating), region_to_coarse_algorithm (e.g. covariate_sum_shift).

GTAP/CGE stage config

All stage-prefixed in multi-stage projects: gtap_model_label (was v1 cge_model_label), gtap_aggregation_label (was aggregation_label), gtap_cmf_template, gtap_shocks_path, gtap_shock_config. In v2, gtap_shock_config additionally carries the per-header time semantics (rate / level / difference) that drive shock harmonization — see Exogenous shocks in the harmonization section.

Workflow control

column type default semantics
run bool 1 Include this row in execution. Replaces commenting-out or deleting rows.
ignore_dependencies bool 0 Skip validation of / waiting on cross-scenario upstream artifacts (assume present or regenerate). This is the boolean that v1 smuggled into scenario_type='baseline_ignore_dependencies'.

Unknown columns

Columns not in this spec are allowed: they are typed by the suffix rules below and set as ProjectFlow attributes. This is the extensibility valve so projects extend the schema instead of forking it.

The canonical clock and model-clock harmonization

Every scenario has exactly one canonical clock: p.base_year (a single int), p.years (a sorted int list), and the derived p.previous_years (a dict mapping each canonical year to the adjacent year toward base_year in its chain). These live on the ProjectFlow object, are set once at scenario load, and are never mutated by stages or iterators. Everything project-wide — cross-stage data exchange, output-directory years, comparison and reporting tasks — speaks this clock.

Any model wired into the project may have different data availability or a different natural cadence. Each such model declares a stage clock (<stage>_base_year, <stage>_years or <stage>_timestep) plus a harmonization policy (<stage>_interpolation, <stage>_extrapolation). A stage with no time columns runs on the canonical clock — the common case.

The motivating configuration (an NGFS-style GTAP+SEALS project):

clock base year run years why
canonical 2020 2030 2040 2050 anchored to observed ESA LULC; ESACCI ends 2022, so a directly observed canonical anchor must be ≤ 2022 (a later anchor is possible via Constructed anchors)
gtap 2023 2024 … 2050 (gtap_timestep=1) the GTAP database year is 2023; the recursive-dynamic solve is annual
seals (canonical) (canonical) SEALS is the keeper of the land state; its clock is the canonical clock
MAgPIE (coarse source) 2020 5-year bands harmonized at NetCDF extraction via coarse_interpolation

The exchange rule

Cross-stage artifacts live on the canonical clock. A stage runs internally on its native clock — GTAP’s annual year-on-year .upd chain is untouched — and anything it publishes for consumption by another stage or by reporting is resampled to canonical years using its declared policy, through one shared hazelbean utility (planned hb.resample_years(values_by_year, target_years, method, extrapolation)). Symmetrically, a stage that needs an input at a native year the canonical grid skips resamples the canonical artifact the same way. The same machinery runs at every clock boundary, in both directions: stage outputs → canonical (publishing), canonical inputs → stage-native (consumption), and exogenous sources → stage-native (shock files, coarse NetCDFs — see below). Task code never hand-rolls year interpolation: the three duplicated linear-interpolation blocks in gtappy_tasks.py (~2898, ~3313, ~3825) and the hardcoded decadal interpolation in the BtC extraction path collapse into this utility. Native-clock artifacts stay in the stage’s own output tree — canonical resampling adds files, it never deletes the annual detail.

How the awkward anchors resolve:

  • GTAP 2023 vs canonical 2020. GTAP’s base year is a stage fact, not the project’s. Canonical years above 2023 are sampled directly from its annual grid (a validated superset). If a task requests GTAP values at canonical years at or below 2023, the default gtap_extrapolation=hold supplies the 2023 base values — an explicit, declared assumption of no change over 2020→2023, instead of a silent one.
  • MAgPIE 2020 / LUH2 2015. Calendar alignment of the NetCDF time axis stays with the existing time_dim_adjustment mechanism (orthogonal to harmonization). Missing years on the aligned axis are filled per coarse_interpolation — default linear between the neighboring bands. This also fixes extract_global_netcdf, which today silently skips requested years the source lacks.
  • ESACCI ends 2022. Not hardcoded anywhere: the canonical base_year must be a year the anchoring observed row can supply (base_year_lulc_path must exist), which is where the ≤ 2022 constraint enforces itself. A future ESA release moves the anchor by editing one CSV cell — or the project constructs a later anchor and makes it canonical (see Constructed anchors).

Exogenous shocks (GTAP baseline drivers)

The GTAP baseline requires exogenous driver shocks (GDP, population, labor) supplied in a .har whose headers come in distinct time families — annual growth rates (OGP/POP/LAB), decadal (DGP/DYP/DYL), cumulative-from-base, and duplicates. Today a run year missing from the shocks file freezes the solve mid-run, and single-step runs flip between header families by special case (gtappy_tasks.py:988, enabled only for len(years)==1 and year ∈ {2030, 2040, 2050}). Under v2 the shocks file is treated like any other clocked source:

  • A shock-preparation step, part of the stage’s base-year preparation, resamples the source shocks onto the stage’s resolved chain and writes a derived .har that matches gtap_years exactly. After that, the CGE loop can never request a year the file lacks.
  • Growth rates compound; they never interpolate linearly. Splitting a decadal shock into annual steps takes the geometric root (1+G)^(1/10) − 1; aggregating annual shocks into a multi-year step (including a singleshot 2023→2050 jump) takes the running product of (1+g_t) minus one. This is the compound method in the interpolation vocabulary.
  • The method is per header, not per file: most drivers are ratio growth rates, some are levels, and at least one (gdppc) is a difference rather than a ratio. gtap_shock_config maps header → semantics (rate / level / difference), and the resampler picks compound, linear, or difference-scaling accordingly.
  • Coverage validation includes shock sources: at load, the years obtainable from gtap_shocks_path under the declared per-header semantics are checked against the resolved stage chain. A gap is a load-time error naming the header and the year — replacing today’s silent freeze.

Coupled stages: canonical years are the barriers

For one-way pipelines the scheduler may run stage by stage. For two-way coupled runs — e.g. GTAP solving annually while SEALS and InVEST recompute ecosystem-service shocks every fifth year — it iterates canonical years outermost: between consecutive canonical years, each stage runs its native sub-chain (GTAP: five annual steps; SEALS/InVEST: one step), then exchange artifacts are published and consumed at the barrier before any stage proceeds. The canonical clock is therefore not just the reporting grid; it is the synchronization grid.

Execution is realized through ProjectFlow’s existing nested-iterator machinery rather than a bespoke scheduler: the nesting order of the iterators is what distinguishes one-way (stage-outer) from coupled (canonical-year-outer), the barrier is the outer iteration boundary, and the feedback is task order plus a file on disk. The iter_stage_runs sketch below describes what one iterator level yields, not a standalone scheduler. See Model specs and iterator-driven staging for the full execution design.

A fast test case with years=2050 degenerates to a single barrier — GTAP takes one step on a compounded 2023→2050 shock, SEALS downscales the 2050 land use, InVEST scores it — with no special-cased code path. Note that shrinking years does not silently shrink a declared stage cadence: a project with gtap_timestep=1 must also blank that cell to make GTAP single-step, because a declared native clock is never discarded implicitly. The idiomatic way to keep the full run and the fast test side by side is two rows with run flags, not cell surgery.

Constructed anchors

The canonical anchor does not have to be a directly observed year — it can be produced by an upstream row. The motivating case: GTAP’s database year is 2023, GTAP data is hard to move, but ESACCI ends at 2022 and ESA is easy to extend. Rather than carrying a permanent gtap stage clock, the project can construct a 2023 LULC anchor and make 2023 canonical:

  1. observed_esa_2022 — a true observed row (base 2022, real ESA map, observed_years=2020 2022). Calibration and validation reference only these genuinely observed years.
  2. extend_anchor_to_2023 — an ordinary bau row anchored on it with years=2023: a mini-SEALS run downscaling the assumed-baseline coarse trajectory’s 2022→2023 change onto ESA (the 2023 slice of 2020/2025 bands comes from coarse_interpolation).
  3. anchor_2023 — an observed row with base_year=2023, depends_on=extend_anchor_to_2023, and base_year_lulc_path templated (cat-ears) onto row 2’s output.

Real scenarios then anchor on anchor_2023, and gtap_base_year empties out — GTAP simply runs the canonical clock. Two validation relaxations make this legal: observed rows may carry depends_on, and base_year_lulc_path must exist at load or be a templated product of a depends_on row (checked once that row completes).

Provenance stays visible instead of buried in a side script: use lulc_src_label=esa-extended (or tags=constructed_anchor), and put the assumed trajectory in the extend row’s counterfactual_label so the construction assumption appears in output paths. Two consequences to state honestly: every scenario, including policy rows, embeds the extension-trajectory assumption in its anchor (it cancels in compared_to differences but not in absolute levels), and a constructed anchor must not depend on stages it feeds — the DAG validator rejects the cycle; here the construction uses only the coarse trajectory and ESA.

To cache the artifact instead of re-running it, provision the raster to base data and set the extend row run=0 — path resolution through get_path() plus ignore_dependencies semantics make caching a policy choice, not an architecture fork. When a real observed year ships, swap one row and delete two.

ProjectFlow attribute contract

Canonical — set once at scenario load, immutable thereafter:

attribute type meaning
p.base_year int canonical anchor (renamed from p.key_base_year; always a bare scalar)
p.years list of int canonical simulated/reported years
p.previous_years dict int→int each canonical year → the adjacent year toward base_year in its chain; the first year of each chain maps to base_year. A dict rather than a parallel list so hindcast chains and sparse grids cannot drift out of alignment.
p.observed_years list of int observed/historical years from the anchoring row

Stage-resolved — set by the iterator when entering a stage, equal to the canonical values when the stage declares no clock:

p.stage_label, p.stage_base_year, p.stage_years, p.stage_previous_years

Per run step — names unchanged from today, so the bulk of per-year task code (path composition from str(p.year), ..._{year}_{previous_year}_ha_diff filenames) does not change:

p.year, p.previous_year, p.time_direction

The scattered idiom previous_year = p.key_base_year if c == 0 else p.years[c - 1] — recomputed today in at least six places across seals_main, seals_process_coarse_timeseries, gtappy_tasks and ecosystem_services_tasks — becomes a read of p.previous_years[year] (or p.stage_previous_years[year] inside a stage). The key_base_years_as_list() coercion helper and the scalar/list special-casing of key_base_year in the parser are deleted outright: *_year columns are always scalar ints, *_years columns always int lists.

Load-time clock validation

  1. <stage>_years never contains the stage’s resolved base year; <stage>_years and <stage>_timestep are mutually exclusive.
  2. Coverage: every canonical year must be obtainable from every enabled stage under that stage’s declared interpolation/extrapolation policy, and every resolved stage year must be obtainable from the stage’s declared exogenous sources (e.g. every gtap year from gtap_shocks_path under its per-header semantics). With none/error policies an uncoverable year is a hard load-time failure naming the stage (or header) and the year — instead of a mid-run KeyError, a silently missing band, or a frozen GEMPACK solve.
  3. interpolation/extrapolation/coarse_interpolation values are a closed vocabulary; unknown methods are a load-time error (new methods get added to hazelbean, not to project task code).
  4. A bau row may have empty years only when it resolves to base-year preparation work alone (e.g. an initial CGE equilibrium at gtap_base_year); such rows exist to be depends_on targets.
  5. observed rows may carry depends_on; base_year_lulc_path must exist at load or be a cat-ears-templated product of a depends_on row (constructed anchors), in which case its existence is checked when that row completes.

Hindcasting (backward years)

years may contain any integers other than base_year (listing base_year itself is a load-time error — it is the anchor, never a simulated step). The parser splits years into two chains:

  • forward chain: years greater than base_year, ascending;
  • backward chain: years less than base_year, descending (closest-to-base first).

previous_year is defined as the adjacent year toward base_year in this chain — base_year for the first element of each chain, otherwise the prior chain element. The p.previous_years dict is exactly this mapping materialized, computed once per clock (canonical and per stage). Allocation still computes state(year) − state(previous_year) exactly as today; in the backward chain previous_year is chronologically later than year, and the signs of the changes fall out naturally.

Each run step carries a derived time_direction ∈ {forward, backward} so tasks that genuinely care (discounting, time-dimension slicing) can branch. Canonical serial order per scenario per stage is the full forward chain, then the full backward chain; the two chains are independent and may be parallelized. Cross-stage ordering of these runs (one-way pipelines vs. coupled feedback) is defined by the barrier rule in the harmonization section — canonical years are the synchronization points.

The iteration contract the hazelbean implementation must satisfy:

def iter_stage_runs(scenario_set, stage):
    for s in scenario_set.enabled_topo_sorted():        # run==1, ordered by depends_on DAG
        base, years = s.stage_clock(stage)              # stage columns else canonical clock;
                                                        # <stage>_timestep expanded here
        fwd = sorted(y for y in years if y > base)
        bwd = sorted((y for y in years if y < base), reverse=True)
        for chain, direction in ((fwd, 'forward'), (bwd, 'backward')):
            prev = base
            for y in chain:
                yield RunStep(scenario=s, stage=stage, year=y,
                              previous_year=prev, base_year=base,
                              direction=direction)
                prev = y

Base-year preparation jobs run once per unique anchor before any RunStep — from observed rows for the canonical anchor, and per stage for a stage anchor that differs from it (e.g. the initial GTAP equilibrium solve at 2023).

Parsing rules (implementation contract)

  1. Read with pd.read_csv(path, encoding='utf-8-sig', dtype=str, keep_default_na=False). An empty string means None; the empty cell is the only null representation (no none/NA/nan literals — this kills the 'nan'-string checks in the v1 parsers).
  2. Comment rows: a scenario_label starting with # skips the whole row.
  3. Lists are single-space-separated within a cell (Excel-safe, never commas). Repeated spaces collapse; leading/trailing whitespace is stripped.
  4. Typing is by column-name suffix, never value sniffing (applied after stripping a registered stage prefix):
    • *_path → path; *_paths → path list
    • years, *_years → int list; *_year → int (always a bare scalar — the v1 key_base_year scalar/list asymmetry is gone)
    • *_timestep → int
    • *_interpolation, *_extrapolation → enum string (closed vocabulary, validated at load)
    • run, ignore_dependencies, *_enabled → bool (1/0/true/false, case-insensitive)
    • depends_on, compared_to, tags, *_labels, dimensions → string list
    • everything else → string.
  5. Stage prefixes are resolved only against the registered stage list (p.scenario_stages).
  6. Paths use forward slashes, always (see the slashes rule in conventions; v1 files containing lulc\esa\... are normalized by migration). Paths resolve through ProjectFlow’s get_path() search order (project input dir → base_data → …).
  7. Templating uses hazelbean cat-ears tokens, <^token^>, in two phases: tokens naming already-defined project attributes substitute at row load; run-namespace tokens (year, previous_year, base_year, direction, stage, scenario_label, plus any row field) substitute per run step, just before use. A token still unresolved when its value is consumed is a hard error — no silent pass-through (this replaces v1’s leave_ref_path_if_fail=True fallback).
  8. Load-time validation: unique labels; closed scenario_type vocabulary; observed_reference_label points at an observed row; depends_on/compared_to reference existing labels; the dependency DAG is acyclic; base_year does not appear in years; observed rows have empty years; policy rows have non-empty compared_to; plus the clock validations in the harmonization section (stage years exclude the stage base year, <stage>_years/<stage>_timestep mutual exclusion, canonical-coverage check under declared policies).
  9. Legacy detection: the presence of a key_base_year column or any scenario_type='baseline' row means the file is v1 — the parser refuses it and points at the one-shot migrator (planned as hb.scenarios.migrate_v1_to_v2(csv_path)). There is deliberately no dual-mode parsing.

Migration from v1

SEALS standard_scenarios.csv (27 columns)

v1 v2
scenario_type=baseline row scenario_type=observed, years emptied
key_base_year base_year
base_years observed_years
baseline_reference_label observed_reference_label
comparison_counterfactual_labels compared_to
calibration_parameters_source calibration_parameters_path
backslash paths (lulc\esa\...) forward slashes
(new) run=1, ignore_dependencies=0, empty depends_on and tags
everything else unchanged

NGFS/PNAS variant (35 columns)

All of the above, plus:

v1 v2
aggregation_label gtap_aggregation_label
cge_model_label gtap_model_label
cmf_template, shocks_path, shock_config gtap_cmf_template, gtap_shocks_path, gtap_shock_config
key_base_year (CGE anchor, 2023), years (annual CGE years) gtap_base_year=2023, gtap_timestep=1 (or an explicit gtap_years list)
seals_key_base_year (2020 observed anchor) plain base_year (the canonical anchor)
seals_years plain years (the canonical clock; SEALS declares no stage clock)
shock_config=none literal empty cell

Row surgery:

  • Add one observed row (anchor 2020, ESA LULC) — v1 had no row representing the actual observed anchor.
  • baseline_ignore_dependencies_initial (type baseline) → ordinary bau row gtap_initial — it is a real CGE run, not data. It carries empty years and an empty gtap clock: a base-year-only equilibrium anchored at gtap_base_year.
  • baseline_ignore_dependencies (type baseline_ignore_dependencies) → bau row with depends_on=gtap_initial, ignore_dependencies=1.
  • Policy rows: baseline_reference_label/comparison_counterfactual_labels pointing at the pseudo-baseline → depends_on=bau, compared_to=bau, observed_reference_label=<the observed row>.
  • Any scenario_type=stress_test-style values → policy plus tags=stress_test.

gtappy template variant

v1 v2
scenario_index dropped (label is the key)
*_short_label columns dropped, or folded into the canonical label columns at migration
baseline_reference_year base_year
variable-extraction / output-filter columns out of scope for scenario definitions — they stay in gtappy’s outputs.csv; a genuinely per-scenario override becomes a gtap_-prefixed column

Breaking changes

  1. The p.key_base_year attribute disappears (→ p.base_year); task code reading it must rename (mechanical: e.g. seals_main.py:109, 540, 562, 592).
  2. scenario_type outside {observed, bau, policy} is rejected.
  3. Stage-override columns require stage registration (p.scenario_stages).
  4. Backslash paths are invalid.
  5. The duplicated v1 parsers (seals_utils.assign_df_row_to_object_attributes, gtappy_utils.assign_df_row_to_object_attributes) are slated for deletion once the hazelbean parser exists.
  6. p.seals_years / p.seals_key_base_year and the override tail in seals_utils.py:598-607 (which clobbers p.years for non-baseline rows) disappear — SEALS either runs the canonical clock (typical) or declares a seals stage clock.
  7. Inside stage task code, reads of p.years / p.key_base_year that mean “the clock I am currently iterating” become p.stage_years / p.stage_base_year (set by the iterator); p.year and p.previous_year keep their names. key_base_years_as_list() (seals_process_coarse_timeseries.py:22) and the scalar/list special case for key_base_year in hazelbean/assign_to_object.py:_parse_year are deleted.
  8. Hand-rolled year interpolation in task code is disallowed: the three duplicated blocks in gtappy_tasks.py and the hardcoded decadal BtC interpolation are replaced by hb.resample_years, and the singleshot header-family switch (gtappy_tasks.py:988) is absorbed by the shock-preparation step.

Model specs and iterator-driven staging — the execution layer

This spec builds on Scenario definitions v2 above (the CSV instance layer) and specifies the model-schema and execution layers above it. The canonical parser, the model-spec registry, and the iterator builder will live in hazelbean next to project_flow.py.

The three layers

The devstack has been carrying three concerns in one place (per-project CSV parsers plus scattered task code). v2 separates them:

layer what it is authored by where it lives
model spec a model’s typed inputs, outputs, clock, and iteration role — the schema for one model (SEALS, GTAP, InVEST-ES) developers, in Python each model’s repo; registry in hazelbean
scenario definitions the values for a particular project — one row per scenario, the canonical clock researchers, in Excel/CSV each project’s scenario_definitions.csv
task tree the executable ProjectFlow — nested iterators built from the specs and the canonical clock the framework project_flow.py (existing engine)

InVEST (NatCap) formalizes only the first layer — a ModelSpec of typed Input/Output objects plus an execute(args) that returns an output registry. We adopt that shape and extend it with the two things InVEST leaves to the user: a clock (a model runs across time, not once) and an iteration role (how the model becomes one or more levels of the ProjectFlow iterator tree). The scenario layer and the harmonization semantics are specified above; this part is the model spec and how the engine consumes it.

What a model spec is

A model spec is one declarative object per model, registered before the run:

p.register_model(seals.MODEL_SPEC)      # replaces the bare p.scenario_stages = ['gtap','seals']
p.register_model(gtappy.MODEL_SPEC)
p.register_model(global_invest.ES_MODEL_SPEC)

Registering a model both declares a stage (its stage_label becomes the <stage>_ column prefix in the CSV) and hands the framework everything it needs to type-check inputs, resolve the model’s clock against the canonical clock, build its iterator level(s), and wire its outputs to downstream models.

Fields

ModelSpec(
    stage_label   = 'seals',            # the <stage>_ CSV prefix; unique
    model_id      = 'seals_downscaling', # globally unique, package-ish
    inputs        = [...],               # typed Input objects (below)
    outputs       = [...],               # typed Output objects — the exchange registry
    clock         = ClockSpec(...),      # base-year source, cadence, harmonization
    iteration     = IterationSpec(...),  # how this model maps to iterator levels
    execute       = seals_main.allocations,  # the callable(s) the engine wraps
    validate      = None,                # optional; defaults to spec-driven validation
)

Typed inputs (replacing suffix typing for known columns)

v2’s suffix rules (*_path, *_years) stay as the fallback for unknown columns — the extensibility valve. But a model’s known columns are declared with types, which is strictly better: it removes the reason the v1 calibration_parameters_source had to be renamed to _path just so the suffix rule could see it. The type hierarchy is small and domain-specific (the InVEST hierarchy adapted — we need years and correspondences, not pint units):

  • PathInput(required, must_exist, templatable) — resolves through get_path(); must_exist may be deferred (see Deferred inputs).
  • YearInput — a single int (a base year). Always scalar; the v1 key_base_year scalar/list asymmetry is gone.
  • YearListInput — a space-delimited int list (years, observed_years).
  • RasterInput, VectorInput(geometry_types, id_field), CSVInput(index_col, orientation) — spatial/tabular, carrying the validation each needs.
  • CorrespondenceInput — a reclassification csv (src_id/dst_id/src_label/dst_label or multicorrespondence).
  • LabelInput, LabelListInput — scenario references (observed_reference_label, depends_on).
  • EnumInput(options) — closed vocabulary (scenario_type, *_interpolation).
  • NumberInput, BoolInput, StringInput — scalars.

Each input carries id, about, required (bool or a string expression evaluated against the row, e.g. "scenario_type == 'policy'"), and a validate(value, row) -> error | None. seals_api_input.md is the human-facing rendering of a spec; the spec object is the machine-facing source of truth, and one can generate the other.

Typed outputs (the cross-stage exchange registry)

This is the piece that makes model coupling concrete. Each model declares its outputs by id, with a templated path and the canonical years at which they exist:

outputs = [
    RasterOutput(id='lulc_projected',
                 path='<^cur_dir^>/lulc_<^year^>.tif',
                 per='canonical_year'),
    VectorOutput(id='regional_luc',
                 path='<^cur_dir^>/luc_<^year^>.gpkg',
                 per='stage_year'),
]

A downstream model consumes an upstream one by output id, never by reconstructing a path — the fragile string-surgery that seals_process_coarse_timeseries does today (cur_dir.replace('/'+year+'/', ...)) is replaced by p.resolve_output('seals', 'lulc_projected', year). This mirrors InVEST’s decision to have execute() return an id→path dict, and it is what lets the harmonizer resample another model’s named output onto the clock it needs.

The clock spec

Attaches the harmonization design from the scenario-definitions spec above to the model rather than to loose CSV columns:

ClockSpec(
    base_year   = 'canonical',          # or 'own' (reads <stage>_base_year)
    cadence     = 'canonical',          # or 'timestep' (<stage>_timestep) or 'explicit' (<stage>_years)
    serial      = True,                 # recursive-dynamic: year N needs year N-1
    interpolation   = 'linear',         # how published outputs fill skipped canonical years
    extrapolation   = 'hold',           # how canonical years outside native range fill
    exogenous   = [ShockSource(path_column='gtap_shocks_path',
                               semantics_column='gtap_shock_config')],  # per-header rate/level/difference
)

serial=True marks the recursive-dynamic chain (GTAP’s .upd, SEALS’s year-on-year allocation) — such a level can never be run_in_parallel. The independent forward/backward hindcast chains and independent scenarios can be.

The iteration spec — how a model becomes iterator levels

IterationSpec(
    levels = ['scenario', 'stage_year'],  # the iterator nesting this model contributes
    replacement_names = {                 # attribute names it sets — distinct per level
        'stage_year':  ['year', 'previous_year', 'time_direction'],
    },
    zones = 'allocation_zones',           # optional inner parallel decomposition (existing)
)

The replacement_names matter because ProjectFlow’s iterator_replacements is a single project-level dict rebuilt by each iterator level’s function. Two levels that both wrote p.year would collide; the canonical-year level writes p.canonical_year while a stage-native sub-year level writes p.year. This is exactly the p.year (per-step) vs p.canonical_year/p.stage_year (stage-resolved) split in the attribute contract above.

How the task tree is built from specs

The engine already supports the primitive we need: nested iterators are in production today. allocation_zones_task is created with parent=p.allocations_task in seals, gtappy and gtap_invest — a two-level scenario→zone nest. Each level’s function repopulates the global iterator_replacements for its own dimension; execution order and the per-iteration copy.copy(self) freeze keep the levels independent. The proposal extends this proven depth-2 pattern to depth 3–4.

A generic builder, p.build_stage_tree(coupling=...), reads the registered specs, the canonical clock, and one project-level coupling mode, and emits the iterator tree. The coupling mode is the single knob that answers both of the motivating cases, because the nesting order of the iterators is what distinguishes one-way from coupled:

Case 1 — one-way (coupling='pipeline'): GTAP annual to 2050, then SEALS every 5 years

Stages are sibling iterator subtrees; within each, the stage is the outer level and its native year the inner:

scenario iterator            (topo-sorted by depends_on; parallel-ok)
├── gtap subtree
│   └── gtap_year iterator    (annual 2024..2050; serial)
└── seals subtree
    ├── seals_year iterator   (5-yearly 2025..2050; serial)
    └── allocation_zones      (parallel; existing inner nest)

GTAP runs its whole annual chain and publishes regional_luc at every year; SEALS samples that output at its 5-yearly canonical years (an exact subset — no interpolation needed). This is the multi_stage_ngfs pattern and needs nothing the engine can’t already do.

Case 2 — coupled feedback (coupling='coupled'): GTAP 5 yrs → SEALS+InVEST make shocks → back to GTAP

The canonical year is the outer iterator and the stages are serial children; the barrier and the feedback both fall out of the tree for free:

scenario iterator                     (parallel-ok across scenarios)
└── canonical_year iterator           (2025 2030 ... 2050; serial — sets p.canonical_year)
    ├── gtap_segment   (child 1)      (gtap_year sub-iterator: the 5 annual steps to this canonical year; sets p.year)
    ├── seals          (child 2)      (downscales this canonical year; + allocation_zones parallel)
    └── invest_es      (child 3)      (scores ES; writes next segment's shock har)
  • The barrier is the outer iteration boundary — canonical_year cannot advance to 2030 until all three children of the 2025 iteration finish. Free from the existing engine (children run to completion within an iteration).
  • The feedback is execution order + a file on disk — invest_es writes the shock har during the 2025 iteration; gtap_segment reads it during the 2030 iteration because children run in tree order (the non-parallel path is literally for child in task.children: run_task(child)). No new cross-scenario depends_on granularity is needed, because the coupling is within one scenario’s iterator subtree.
  • Feedback flows through disk, never through p — the engine’s own comment warns that parallel project_copy mutations “can’t be rejoined.” The shock har is the handoff, which is already how these models communicate. Consequently the coupled inner levels are serial; only the scenario level parallelizes.

The same three model specs produce case 1 or case 2 purely by the coupling mode flipping the nesting order. A researcher’s fast test (years=2050) degenerates to a single outer iteration with no special-casing.

Clock resolution, harmonization, deferred validation

These come from the scenario-definitions spec above, now anchored to the model spec:

  • Resolution — for each stage the builder computes (base_year, years) from its ClockSpec (canonical unless it declares own/timestep/explicit) and materializes previous_years per level. The iterator gets a flattened parallel previous_year list built from that dict — exactly what seals_main already assembles by hand.
  • Harmonization — all cross-clock resampling goes through one utility, hb.resample_years(values_by_year, target_years, method, extrapolation), semantics-aware: levels interpolate, rates compound (geometric root to split, product to aggregate), differences scale. This replaces the three duplicated interpolation blocks in gtappy_tasks.py, the hardcoded decadal BtC interpolation, and the silent year-skip in extract_global_netcdf.
  • Load-time validation — the spec makes coverage checkable: every canonical year must be obtainable from every registered model under its declared policy, and every resolved stage year from its declared exogenous sources. A gap is a named load-time error, not a mid-run KeyError or a frozen GEMPACK solve.
  • Deferred inputs — an input whose must_exist is deferred (a constructed-anchor LULC, or a coupled shock har that a later barrier produces) is not checked at load; it is checked when its producing task completes. This is the one place case 2 genuinely needs the declarative layer: the segment-2 shock file does not exist at load, so gtap_shocks_path’s existence check is deferred to the barrier, exactly as a constructed anchor’s base_year_lulc_path is deferred to its depends_on row.

What is reuse vs. new work

Honest accounting, because the point of anchoring on iterators is to minimize new engine code:

Reuse (exists today): the iterator engine, nested iterators (proven at depth 2), iterator_replacements, per-iteration project_copy freezing, serial-vs-parallel per level, the assign_to_object value parser, and initialize_definitions_csv.

New, but pure data (no engine changes): the ModelSpec/ClockSpec/ IterationSpec dataclasses; the registry (p.register_model); clock resolution and previous_years materialization; the coverage validator.

New engine-adjacent work, to prototype and test:

  1. p.build_stage_tree(coupling) — the generic builder that nests specs into iterators. The depth-3–4 nesting is an extension of a working depth-2 pattern, but must be tested: attribute-name collisions across levels, and the global iterator_replacements rebuilt correctly at each level.
  2. hb.resample_years — hazelbean has spatial interpolation only; this temporal, semantics-aware utility is genuinely new.
  3. Deferred must_exist validation — checking a templated input at producer-completion instead of at load.
  4. The output registry (p.resolve_output(stage, id, year)) replacing path string-surgery.

Intentionally unchanged: per-step task code (str(p.year) path composition, ..._{year}_{previous_year}_ha_diff filenames), because p.year/p.previous_year keep their names and meanings.

Ownership and migration

Spec machinery (dataclasses, registry, builder, resampler, validator) lives in hazelbean next to project_flow.py. Spec content lives in each model’s repo: seals.MODEL_SPEC, gtappy.MODEL_SPEC, global_invest.ES_MODEL_SPEC — the same ownership split as the rest of the devstack. seals_api_input.md becomes the rendered view of seals.MODEL_SPEC; the stub parser in seals_api_parsing.py (and its generate_scenarios_csv_and_put_in_input_dir inverse) is superseded rather than revived. A first useful deliverable is “generate a valid v2 template CSV from the registered specs” (generate_scenarios_csv_from_model_spec), which replaces the hand-maintained template writer and gives new users the runnable example the SEALS input doc already promises. Note: seals’ runtime generate-if-missing branch was retired in 2026-08 in favor of tracked input_template/ CSVs, so this deliverable is a restoration of that capability with a schema guarantee, not a new idea.

Future implementation

The canonical parser, validator, migrator and RunStep iterator belong in hazelbean next to project_flow.py, reusing cat_ears.py for templating, plus three harmonization pieces: a semantics-aware hb.resample_years (the single temporal resampling utility — levels interpolate, rates compound, differences scale; hazelbean currently has spatial interpolation only, nothing temporal), a coarse_interpolation-aware extract_global_netcdf (which today silently skips requested years the source lacks), and the shock-preparation step that writes a stage-chain-aligned .har from Erwin-style driver files. Key v1 code to replace when implementing: seals/seals_utils.py:586-724 (row→attributes parsing, including the seals_years override tail), seals/seals_main.py:1400-1560 (scenario/year iteration), the gtappy/gtap_invest copies, and the interpolation blocks in gtappy_tasks.py.

Additional ideas

1. One row per solved counterfactual (retire shock_config)

Every counterfactual the CGE model solves is an explicit row in scenarios.csv — no mechanism may graft, suffix, or otherwise synthesize counterfactuals at run time. Decomposition variants of a policy (below_2c, below_2c-noes) are sibling rows sharing the same exogenous/climate/model labels, distinguished by the counterfactual processing-step suffix (hyphen + 4 chars, already in the scenario-naming rules). Pass ordering is expressed with depends_on, not loop order in task code. Rationale: v1’s shock_config produced counterfactuals (*_cc_es) that existed in results but not in the CSV, breaking the “scenarios.csv is authoritative” contract and forcing downstream consumers to discover scenarios by disk-scanning and suffix-matching — a documented source of silent-empty-figure bugs.

2. A first-class static clock

clock_type (or an empty years with a defined meaning): static declares a comparative-static solve — base data plus shocks, no elapsed time. A static scenario has no year dimension: no year path segment in outputs, no year column joins across projects, and appending years to it is a load-time error. Rationale: v1 forces a fake bookkeeping year (e.g. 2024) into every path and figure; the number is economically inert but invites temporal misreading, permits silent shock-compounding if someone extends years, and creates meaningless joins against projects where the same year is real. Note the closure caveat: whether a solve is truly static or one recursive-dynamic period is a property of the CMF/closure (cmf_template), which the clock declaration does not override — the spec should require the two to agree.

3. Shock composition is data, not code

Which shocks a row applies is expressed in columns — a *_path column per shock channel (hydration is name-driven) and a shocks-snippet path for the CMF statement — never as tokens interpreted by task code, in-code label-to-file correspondence dicts, or derived filename conventions. The two path kinds are distinct and both required: shocks_path names a CMF text fragment (appended verbatim, written in final cat-ear form); <channel>_shock_path names the data a project task converts to the HAR the fragment references.

4. Stage ownership of scenario columns (the layering rule)

A model stage’s initializer may hydrate and validate only its own column group, and no stage’s initializer may hard-wire another stage’s post-processing (the standing violation: gtappy’s scenario init unconditionally running seals’ set_derived_attributes, which forces LULC-raster and correspondence-file dependencies onto pure GTAP projects). What a project includes is declared by its task tree — build_task_tree is already the canonical answer to “what is this pipeline” — and each stage present validates its required columns at load time via its MODEL_SPEC, failing with a named column list rather than a downstream AttributeError. Machinery (registry, validator, template generation) lives in hazelbean; content (each stage’s spec and hooks) lives in each model’s repo. Corollary: a scenarios CSV carries only the column groups of stages its project uses — ballast columns for absent stages are a v1 compatibility artifact, not a convention.

5. Downstream consumers read the CSV, not the disk

Reporting and visualization derive the scenario/counterfactual universe from scenarios.csv (labels + comparison_counterfactual_labels), never by globbing result directories or pattern-matching label suffixes. Suffix conventions are for humans and filenames; they are not an API.

6. Solve steps are explicit tasks (small, run-file anatomy bullet)

A pipeline whose generation step defers execution to a dispatch (CMF generation vs GEMPACK solve) must carry the solve as its own named task in the tree — the tree should read as the complete causal chain, and a missing solve step should be visible in build_task_tree, not discovered as a missing-results crash three tasks downstream.