diff --git a/AGENTS.md b/AGENTS.md index a4f627d..8d748be 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,8 +16,6 @@ Worldwide PhenoCam EFAST feasibility screening. Human summary: [`README.md`](REA | `data/` | Manifests, per-site caches, screening outputs (large; mostly generated) | | `index.html`, `common.js` | Static QA viewer (`make serve` from workspace root) | -Workspace orchestration: [`../AGENTS.md`](../AGENTS.md). - --- ## Where to work diff --git a/README.md b/README.md index 36503d5..1e790fa 100644 --- a/README.md +++ b/README.md @@ -1,38 +1,44 @@ -# Worldwide PhenoCam EFAST screening and fusion +# EFAST fusion with phenocam validation. -Screen the global [PhenoCam Network](https://phenocam.nau.edu/) for sites where EFAST Sentinel-2 / Sentinel-3 fusion is likely to work: enough PhenoCam `gcc_90`, seasonal signal, and S2/S3 coverage for a calendar year. +End-to-end pipeline from the global [PhenoCam Network](https://phenocam.nau.edu/) to validated EFAST Sentinel-2 / Sentinel-3 fusion results. The five numbered steps cover site selection, Sentinel data acquisition, BtI/ItB fusion, and accuracy metrics, all feeding a static web QA viewer. -Agent-oriented detail: [`AGENTS.md`](AGENTS.md). +--- + +## Pipeline overview + +| Step | Script | What it does | +|------|--------|--------------| +| 1 | `1-phenocam.py` | Download PhenoCam metadata and `one_day_summary` GCC CSVs | +| 2 | `2-phenocam-screening.py` | Apply PhenoCam count, SNR, and proximity gates to select feasible sites | +| 3 | `3-sentinel-data.py` | Acquire S2 (Earth Search COG) and S3 OLCI SYN L2 (CDSE OpenEO); prepare REFL, DIST_CLOUD, and composite GeoTIFFs | +| 4 | `4-fusion.py` | Run EFAST BtI (fuse reflectance → GCC) and ItB (fuse GCC directly) for each screened site | +| 5 | `5-metrics.py` | Extract PhenoCam-matched timeseries, compute NSE/RMSE/r baselines and fusion metrics, emit per-site JSON and webapp manifest | --- ## Quick start -From `processing/`: - ```bash uv sync uv run python 1-phenocam.py --evaluation-year 2025 -``` - -### Stepped pipeline (resumable) - -All steps use `--evaluation-year` (default 2025) and optional `--site`. See each script docstring for inputs/outputs under `data/`. - -```bash -uv run python 1-phenocam.py --evaluation-year 2025 uv run python 2-phenocam-screening.py --evaluation-year 2025 uv run python 3-sentinel-data.py --evaluation-year 2025 uv run python 4-fusion.py --evaluation-year 2025 uv run python 5-metrics.py --evaluation-year 2025 - -# single site -uv run python 3-sentinel-data.py --evaluation-year 2025 --site innsbruck -uv run python 4-fusion.py --evaluation-year 2025 --site innsbruck -uv run python 5-metrics.py --evaluation-year 2025 --site innsbruck ``` -Step 3 S3 uses CDSE OpenEO (`SENTINEL3_SYN_L2_SYN`); S2 uses AWS Earth Search COG range reads (no auth). CDSE credentials live in `../.env` at the workspace root (`CDSE_USER`, `CDSE_PASSWORD`). +All steps accept `--evaluation-year` (default `2025`) and `--site` (optional, for single-site runs). Steps 3–5 are resumable — existing output files are skipped. + +```bash +# single site +uv run python 3-sentinel-data.py --evaluation-year 2025 --site ICOSFR-Fon1 +uv run python 4-fusion.py --evaluation-year 2025 --site ICOSFR-Fon1 +uv run python 5-metrics.py --evaluation-year 2025 --site ICOSFR-Fon1 +``` + +### Credentials + +Step 3 S3 download uses CDSE OpenEO (`SENTINEL3_SYN_L2_SYN`). Set `CDSE_USER` and `CDSE_PASSWORD` in `../.env` at the workspace root. S2 uses AWS Earth Search COG range reads (no auth required). --- @@ -41,20 +47,19 @@ Step 3 S3 uses CDSE OpenEO (`SENTINEL3_SYN_L2_SYN`); S2 uses AWS Earth Search CO | Artifact | Step | Role | |----------|------|------| | `phenocam/{year}.json` | 1 | Site list + `sites_dir` pointer | -| `phenocam/{year}/{site}.json`, `{site}_1day.csv` | 1 | Raw API + GCC CSV | -| `phenocam_screening/{year}.json` / `.csv` | 2 | PhenoCam + SNR gate results | +| `phenocam/{year}/{site}.json`, `{site}_1day.csv` | 1 | Raw API payload and GCC CSV | +| `phenocam_screening/{year}.json` / `.csv` | 2 | Gate results (pass/fail per site) | | `sentinel_data/{year}/{site}/prepared/s2/` | 3 | S2 REFL + DIST_CLOUD GeoTIFFs | | `sentinel_data/{year}/{site}/prepared/s3/` | 3 | S3 composite GeoTIFFs | -| `fusion/{year}/{site}/` | 4 | BtI/ItB fused rasters | -| `metrics/{year}/{site}/`, `metrics/manifest.json` | 5 | Timeseries JSON, covariates, webapp manifest | - -The 2025 manifest currently lists **739** cameras with archive overlap; most per-site CSV/JSON files are cached under `data/phenocam/2025/`. +| `fusion/{year}/{site}/bti/`, `.../itb/` | 4 | BtI fused reflectance + GCC; ItB fused GCC | +| `metrics/{year}/{site}/` | 5 | Per-site timeseries, metrics, covariates JSON | +| `metrics/manifest.json` | 5 | Webapp manifest (years + site metadata) | --- ## Web viewer -From the workspace root, `make serve` serves `processing/` at [http://localhost:8000/index.html](http://localhost:8000/index.html). Requires step 5 (`data/metrics/manifest.json`). +`make serve` (from the workspace root) serves `processing/` at [http://localhost:8000/index.html](http://localhost:8000/index.html). Requires step 5 output (`data/metrics/manifest.json`). ---