SARtale Guide

Pipeline

The pipeline in processing/ rebuilds the lake data from NASA's archive. The team runs it; the website only reads what it writes.

Before you start: create the sartale conda environment (Getting started) and save a NASA Earthdata login in ~/.netrc. Never commit the .netrc file or paste its contents anywhere.

Run it#

All commands run from processing/, through one entry point, python -m sartale.run_pipeline:

cd processing
conda run -n sartale python -m sartale.run_pipeline lakes-search   # 1. find NISAR passes over each lake
conda run -n sartale python -m sartale.run_pipeline lakes-fetch    # 2. cache a 6 km window per pass (restartable)
conda run -n sartale python -m sartale.run_pipeline lakes-absence  # 3. re-check the sites with no lake now
conda run -n sartale python -m sartale.run_pipeline lakes-build    # 4. write lakes.json and the lake pictures
conda run -n sartale python -m sartale.run_pipeline lakes-check    # 5. compare radar water with Sentinel-2 water
conda run -n sartale python -m sartale.run_pipeline lakes-freeze   # 6. which lakes are probably frozen

The commands#

CommandWhat it doesReadsWrites
lakes-searchSearches NASA CMR for NISAR GCOV granules over each lake in the standard collection (NISAR_L2_GCOV_V1) and the provisional one (NISAR_L2_GCOV_PROVISIONAL_V1), keeps the standard copy of an acquisition found in both, and prints a coverage report. Exits 1 if no lake has any pass (a catalogue outage)lakes_in.jsondata/lakes/passes.json
lakes-fetchReads a 6 km window from each pass with HTTP range requests and caches it. You can stop it and run it again; it skips cached windows and exits 1 if any pass failedpasses.jsondata/lakes/<lake>/<date>_<track>_<CRID>.npz
lakes-buildRuns the method on the cached windows of the passes passes.json lists, whatever their CRID, then the Attabad gate (below). A cached window of a pass the catalogue no longer lists is ignored, not deleted, so a long-lived cache builds the same as a fresh one; it exits 1 without passes.json. A lake whose status track mixes CRIDs gets a limitthe cache, passes.json, lake_checks.jsonapp/public/data/lakes.json, app/public/data/lakes/<lake>/*.webp
lakes-checkCompares radar water with Sentinel-2 water for --lakes (default: every register lake except sites with no lake now), on the clearest pair inside the open-water season (15 Jun to 30 Sep)the cache, lakes.jsonprocessing/lake_checks.json; the next lakes-build copies it into lakes.json
lakes-absenceRe-checks every site with no lake now with Sentinel-2, all year: the newest scene since the last check (or, for a site with none yet, in the last 400 days) whose lake zone is at least 80% clear of cloud, shadow, snow and ice (pixels within 100 m of cloud or cloud shadow are not clear), with snow or ice on no more than 1% of it. Two such scenes at least 5 days apart, both above the site's threshold, make it Lake may be forming (one is only pending); two below make it No lake now again. The search follows every page of results. When no scene qualifies, or the search fails, the last check stands with the date of the search. The rebuild runs it before lakes-build, so the build carries the new check (and a forming site keeps its radar water layer) in the same refresh; --only-missing, after the build, checks a site new to the registerlakes.json, Earth Searchabsence_checked and the status in lakes.json; lakes-build carries the last check forward while the zone is the same
lakes-freezeWorks out each lake's freeze state (see Method, Frozen lakes): NASA POWER daily air temperature from 1 Sep (one retry), moved to the lake's height, and for a cold lake the clearest Sentinel-2 scene of the last 7 days (read on the lake's radar window from the cache, for its outline or summer radar core). Flags each pass taken while frozen and sets the statuses again. Needs no login; a failed download leaves that lake "not assessed" with its usual status. Sites with no lake now get no freeze state. Runs after lakes-build (which rebuilds the passes without flags)lakes_in.json, lakes.json, the cache, NASA POWER, Earth Searchfreeze, each pass's frozen and the status in lakes.json
lakes-elevationMeasures, once, the surface height of every register lake without one: the 10th percentile of the Copernicus DEM GLO-30 within 1 km of the lake point. The rebuild runs it before lakes-build, so a newly accepted lake gets its height in the same refresh; it needs no login and never fails the run: a lake whose DEM cannot be read keeps no height, its freeze is not assessed, and the next refresh tries againlakes_in.json, Earth Searchelevation_m and elevation_source in lakes_in.json
lakes-diffCompares a previous lakes.json (--old) with the new one pass by pass: prints added and dropped passes per lake, writes them to the GitHub job summary, warns for every dropped pass and every lake whose latest date moved back, and exits 1 if more than a quarter of the previous passes would be dropped. The rebuild runs it against the published filetwo lakes.json filesthe log and $GITHUB_STEP_SUMMARY

Each command takes --help. The paths above are the defaults, and each one can be overridden with a flag: --lakes-in, --passes, --cache, --out or --checks.

The Attabad gate#

lakes-build refuses to call a build good unless the method can tell a real lake from no lake. Attabad is a large, permanent lake; at the Shisper point, the basin of an ice-dammed lake that drained in 2022, there is no lake now. On each shared track, Attabad's water area must be at least twice Shisper's (median ratio ≥ 2), Attabad must have a status, and it must have at least 3 looks in the cloud window. If the gate fails, lakes.json is still written for inspection, but the command exits 1. Pass --allow-gate-fail only while debugging, and never ship that build.

Add a lake by hand#

  1. Add an entry to processing/lakes_in.json (fields in Data files). Every coordinate needs a published source in coord_source.
  2. If the lake has an outline, save it as GeoJSON in processing/outlines/ and set outline to its path relative to processing/.
  3. Run lakes-search, lakes-fetch and lakes-build, then pytest tests -q and npm --prefix app test -- --run.

A lake put forward on the website goes through the same steps after a team member accepts its GitHub issue and lake-issue-accept adds its entry. See Lake requests.

Where the code lives#

ModuleJob
run_pipeline.pyThe command line: every command above and the lake request commands
config.pyEvery method constant, the cloud window and the manual exclusions. No CRID is pinned: each pass keeps its own
lakes.py, lake_catalog.py, catalog.pyThe lake register, the CMR collections and search, picking passes, parsing granule names
remote.py, gcov.pyHTTP range reads of remote HDF5 files, and reading a GCOV window with its mask
radiometry.py, lakewater.pySpeckle box and Lee filter, decibels, the D17 water rule, the screens' dark rule, pass screening and the lake zone
lakestatus.pyEach pass against its own track (z), the alert levels, the message and the status of each lake; demmask.py: the GLO-30 layover and shadow mask of the window median
lakebuild.py, render.pyPuts it together: lakes.json, the pictures and the Attabad gate
lakes_diff.pyThe pass-by-pass comparison behind lakes-diff
lakecheck.pyThe Sentinel-2 comparison
lake_requests.py, lake_issues.py, accept_queue.pyThe request rules (point, name, source, distance), reading issues into requests.json and their comments and labels, and choosing the next waiting accept

Troubleshooting#

  • 401 or a login page from ASF: the Earthdata login in ~/.netrc is missing or wrong, or the ASF application has not been authorised in your Earthdata profile.
  • lakes-fetch exits 1: some passes failed, usually because of the network. Run it again; cached windows are kept.
  • The gate fails after a change to the method: read the per-track ratios it prints. A drop usually means the water rule now misses real water, or reads more of Shisper's empty basin as water.