Installation

There are three types of installations. Basic, which just lets you be a user of Hazelbean (and soon SEALS), Project Installation, which provides a pre-configured SEALS environment that works with project-specific data, and then Developer installation, which lets you make changes to SEALS and the rest of the Earth-Economy Devstack via Pull Requests. Also check out our video playlist for more context.

1. Basic Installation

Hazelbean is the computational core of the devstack and can be installed using condaforge following these steps. See the Installing Git and Miniforge3 video for a walkthrough.

  1. Install git: https://Git-scm.com/downloads
  2. Install Miniforge3 from https://conda-forge.org/download/
    • Install just for your User Account
    • During installation, select yes for “Add Mambaforge/Miniforge to my PATH environment Variable”
  3. Create a new conda environment, activate it, and then install required packages with the following 3 mamba commands. The third command will take quite a while (10+ minutes). The conda init command may not be needed depending on how you configured your shell.
    1. conda init
    2. conda create -n environment_name
    3. conda activate environment_name
    4. mamba install hazelbean

You can now import hazelbean and it will have precompiled binaries already working for your operating system for geospatial operations using ProjectFlow!

2. Project Installation

If someone has given you a project repository that you want to run and it relies on some combination of our devstack tools, but you are not planning on editing any of the tools, a Project Installation is your best approach. See also our video tutorial for Project installation.

A project repository is one that owns a run_<project>.py plus its input_template/ CSVs, and imports a devstack library rather than modifying it. It should carry its own installation steps in its readme; some also include a setup_environment.bat or shell script that builds the environment for you. Note that you need a C/C++ compiler installed, so be sure to read that section below.

If you are starting a project of your own rather than running someone else’s, see Run Templates and copy run_template_downstream_user.py as the seed — it is the worked example of importing a library read-only and adding your own tasks around its task tree.

3. Developer Installation

The Developer Installation installs all (or a subset) of our full devstack. This requires you to have a C/C++ compiler installed (see below). It also specifies canonical file names and structures for where to put the various repositories.

  1. Create a new environment and follow the basic installation steps above.
  2. Remove the packaged hazelbean from the new environment, keeping its dependencies. Which command you need depends on how it got there, so check first with conda list hazelbean and read the Channel column:
    • a conda channel (conda-forge) — conda remove hazelbean --force (NOTE: simple pip uninstalling leaves behind Zombie Folders)
    • pypi — pip uninstall hazelbean. Conda has no record of a pip-installed package, so conda remove fails with PackagesNotFoundError and leaves it in place.
  3. Create the hazelbean Project Folder at C:/Users/<YOUR_USERNAME>/Files/hazelbean/ (PC) or /Users/<YOUR_USERNAME>/Files/hazelbean/ (Mac/Linux)
  4. Clone the desired branch from the hazelbean GitHub repository into your hazelbean Project Folder.
    • git clone -b main https://github.com/jandrewjohnson/hazelbean_dev.git
  5. Navigate to the repo at /Users/<YOUR_USERNAME>/Files/hazelbean/hazelbean_dev/ and then make an editable install of hazelbean with the following command:
    • pip install -e . --no-deps

Add Additional Devstack Repositories as Desired

Now that you have the developer version of hazelbean, you can add additional repositories from our devstack as desired. To do this, using SEALS as an example, follow these steps:

  1. Clone the desired repository, e.g., https://github.com/jandrewjohnson/seals into the proper Project Folder.
    • In the SEALS Project Folder at /Users/<YOUR_USERNAME>/Files/seals/ you will have the Project Repository at /Users/<YOUR_USERNAME>/Files/seals/seals/
    • Navigate to the Project Repsoitory and make an editable install with pip. Make sure you have the right environment activated in your command prompt.
      • pip install -e . --no-deps
  2. Step 1 can be repeated for any of the repositories below (presuming you have read access) - https://github.com/jandrewjohnson/seals - https://github.com/jandrewjohnson/gtap_invest_dev - https://github.com/NatCapTEEMs/global_invest_dev - https://github.com/jandrewjohnson/gtappy_dev - https://github.com/jandrewjohnson/gep_dev

--no-deps is not optional here

Some devstack repositories declare a hazelbean floor — SEALS, for instance, requires hazelbean>=2.0.0. Omit --no-deps and pip will try to satisfy that from PyPI: it uninstalls your editable hazelbean and replaces it with a released wheel. Nothing appears to go wrong at the time. You find out weeks later, when a change you know you made to hazelbean has no effect.

Two things guard against this, and it is worth knowing both:

  • Always pass --no-deps. Install dependencies deliberately, never as a side effect of installing a devstack repo.
  • Keep your hazelbean clone on a branch whose derived version is at or above the floor. Versions come from git tags via setuptools_scm, so a branch that cannot reach the release tag derives a lower version — a clone on develop after a release tagged only on main reports something like 1.8.1.dev59, which does not satisfy >=2.0.0, and pip replaces it even though your code is newer than the release. Back-merging main into develop after each release keeps release tags reachable and closes this off.

Check you are running the code you think you are

After adding each repository, confirm the imports still resolve to your clones rather than to a packaged copy:

import hazelbean as hb
print(hb.__file__)

This must print a path inside your clone (.../Files/hazelbean/hazelbean_dev/hazelbean/__init__.py). If it prints something under site-packages, a packaged copy is shadowing your clone: uninstall it as in step 2 above and redo the editable install. Check here rather than only after installing hazelbean itself — the shadowing is caused by installing the other repositories, so it cannot show up until this point.

4. Per-machine configuration

Installing the code is not quite the whole setup. A few settings differ on every machine — where your lab drive is mounted, which SLURM account to charge — and none of them can be committed, because they are true only of your computer. They live in a single never-committed file:

~/.config/hazelbean/machine.env

Hazelbean loads it the moment you import hazelbean. The format is plain KEY=value lines; a leading export and a trailing # comment are both tolerated, so the same file can also be sourced from a shell. Values are applied with setdefault semantics — a real environment variable always wins — so you can override any of it for one run without editing the file.

Nothing here is required. A machine with no machine.env at all works; you just lose the conveniences below.

The keys that exist

key what it does
HB_SHARED_DATA_DIRS Directories that mirror base_data’s layout and can be read directly — usually a mounted lab drive. get_path searches them after your local base_data and before the cloud bucket, and copies anything it finds into base_data so later runs resolve locally. os.pathsep-separated for more than one.
GTAP_SC_SLURM_ACCOUNT, GTAP_SC_SLURM_PARTITION, GTAP_SC_SLURM_TIME, GTAP_SC_SLURM_CPUS, GTAP_SC_SLURM_MEM How gtappy sizes and charges the sbatch jobs it submits when solving GTAP on a SLURM cluster.
GTAP_SC_WINE_SIF Path to the Apptainer image that runs GEMPACK under Wine on the cluster.

machine.env or parameters.csv?

Both hold per-machine values, and the rule for which is which is simple:

  • If a project’s input_template/ ships the key with a blank value, it belongs in that project’s input/ copy of parameters.csv. The GTAP solve-backend connection settings work this way — vm_ssh_host, vm_disk_prefix, gempack_dir, sc_ssh_host, sc_scratch. They used to be environment variables and deliberately are not any more.
  • Everything else — settings no project varies, that are simply true of the machine — belongs in machine.env.

Many projects need neither. global_invest, for example, has no parameters.csv at all, so running one of its services needs nothing beyond an optional HB_SHARED_DATA_DIRS.

Setting up a shared data root

If you have been given access to a lab drive and mounted it (Google Drive for Desktop, Dropbox), let hazelbean find it:

hb-setup-machine-env            # scan, show what was found, confirm, write
hb-setup-machine-env --print    # show the line without writing anything

It checks the handful of places such a mount can live on your OS, accepts a directory only if it really contains base_data’s top-level folders, and appends the HB_SHARED_DATA_DIRS line for you. Run it once, whenever your situation changes — it is not an install step, and it never overwrites a value you set by hand.

Finding nothing is a normal outcome, not a failure. Without a shared root, get_path falls through to the public cloud bucket, which needs no credentials and works everywhere. That is also the only option on Linux, since Google Drive for Desktop has no Linux client — so cluster machines are expected to have no shared root configured, and pipelines must never depend on one being present.

Check it worked

On a Developer Installation, first confirm you are running your clones rather than a packaged copy — see Check you are running the code you think you are above. That check is worth repeating any time a change you know you made appears to have no effect.

Then confirm data resolution is wired up, without starting a real run:

import os
import hazelbean as hb

p = hb.ProjectFlow(project_dir=os.path.expanduser('~/onboarding_check'))
print(p.shared_data_dirs)   # [] if no shared root is configured -- that is fine
print(p.get_path('cartographic', 'ee', 'ee_r264_correspondence.gpkg'))

A path printed under your local base_data means it worked, whether the file came from the drive or was downloaded from the bucket. If instead it raises, the error lists every root it searched, marks each shared root available or not, and tells you what to do next — so a missing mount is never confused with a missing file.

(The explicit project_dir keeps this snippet runnable from anywhere. Real run files pass project_name instead and let ProjectFlow infer the location from the repo they live in, which needs the script to be inside a git repo — see Run Templates.)

Compiling C/C++ Code

Windows:

  • Option 1: You could go to https://visualstudio.microsoft.com/visual-cpp-build-tools/ and select download build tools.

  • Option 2: Enter the following command in the Terminal: winget install Microsoft.VisualStudio.2022.BuildTools --force --override "--passive --wait --add Microsoft.VisualStudio.Workload.VCTools;includeRecommended" This will launch the build-tools installer (you could do this manually via the MS website if you want, but this ensures you get the right tools).

  • Option 3: Run the install.bat file in the Earth Economy Devestack repo’s root. This just runs the winget command above. - Mac:

Mac/Linux:

  • You can use Xcode to compile the cython files. Most users will already have this installed but if not, follow the directions below.
    • If you don’t have Xcode, you can get it by running xcode-select --install in the Terminal. This command downloads and installs the Xcode Command Line Tools, which includes gcc and clang, the compilers needed to compile C/C++ code on macOS. This is somewhat analogous to the Visual Studio Build Tools on Windows.

Common problems

  • You MUST have administrator rights to your computer.

  • If you’re using Windows PowerShell (instead of the Command Prompt and it isn’t working with Conda, you must initialize conda in powershell)

    • conda init powershell

  • If you don’t add conda to your path, you can do this manually. On PC, you could use the command

    • SETX PATH "%PATH%;C:\Users\<YOUR_USERNAME>\miniforge3;C:\Users\<YOUR_USERNAME>\miniforge3;"
    • (you can do for All Users, but you will need to manually set paths to conda)
      1. If you have an Apple “m1 or m2” chip (a relatively new apple chip, make sure you select the Apple Silicon option). - Install in C:\Users\<YOUR_USERNAME>\miniforge3 (PC) or ~/miniconda3 (Mac)

  • If you get a “Windows Protected your PC”, click more info then Run Anyway.