Foo

README.md

@@ -1,27 +1,30 @@
 # Satellite Data Fusion Pipeline
 
-A Python pipeline for downloading, processing, and fusing Sentinel-2 and Sentinel-3 satellite imagery to generate high-resolution NDVI time series.
+Python pipeline for downloading Sentinel-2 and Sentinel-3 imagery and PhenoCam ground truth, applying NDVI-based cloud pre-selection, fusing sensors with the [EFAST](https://github.com/DHI-GRAS/efast) algorithm, and evaluating fused **Green Chromatic Coordinate (GCC)** time series against PhenoCam `gcc_90`.
 
 ## Features
 
-- **Data Download**: Downloads Sentinel-2 L2A (via AWS Earth Search) and Sentinel-3 OLCI (via OpenEO/Copernicus)
+- **Acquisition** — S2 L2A (AWS Element84 STAC), S3 OLCI L1B (Copernicus OpenEO), PhenoCam midday images and GCC CSV
-- **Cloud Detection**: Identifies cloud-covered images using NDVI analysis
+- **Pre-selection** — Aggressive and non-aggressive NDVI-based cloud screening (plus dark-scene rejection)
-- **EFAST Fusion**: Combines S2 and S3 data using the EFAST algorithm for enhanced temporal resolution
+- **Preparation** — Harmonised reflectance/GCC rasters, distance-to-cloud weights, S3 compositing and optional temporal smoothing
-- **NDVI Calculation**: Generates Normalized Difference Vegetation Index from raw and fused data
+- **Fusion** — EFAST under eight scenarios per site (BtI and ItB × two strategies × σ ∈ {20, 30} days)
-- **Web Visualization**: Interactive web viewer for exploring NDVI time series and imagery
+- **Post-processing** — Crop to valid-data window; NDVI and GCC timeseries at the site
+- **Metrics** — Temporal comparison vs PhenoCam (`metrics.json`); optional Tier-2 withheld-S2 gap validation
+- **Web viewer** — Static HTML dashboard over pipeline outputs (`webapp/`)
 
 ## Installation
 
 ```bash
 pip install -r requirements.txt
-pip install git+https://github.com/DHI-GRAS/efast.git
+pip install git+https://github.com/DHI-GRAS/efast.git   # not on PyPI
 ```
 
-## Configuration
+Create `.env` with Copernicus Data Space credentials:
 
-Set environment variables for Copernicus Data Space authentication:
+- `CDSE_USER`
-- `CDSE_USER`: Copernicus Data Space username
+- `CDSE_PASSWORD`
-- `CDSE_PASSWORD`: Copernicus Data Space password
+
+Python version is pinned in `.python-version` (use `.venv/` locally).
 
 ## Usage
 
@@ -31,54 +34,98 @@ from run import run_pipeline
 run_pipeline(season=2024, site_position=(47.116171, 11.320308), site_name="innsbruck")
 ```
 
-The pipeline processes data in stages:
+`site_position` is always **`(lat, lon)`**. Study sites are listed at the bottom of `run.py`: `innsbruck`, `forthgr`, `pitsalu`, `vindeln2`, `sunflowerjerez1`, `institutekarnobat`.
-1. Download S2/S3 imagery
-2. Generate NDVI from raw data
-3. Detect clouds
-4. Prepare data for fusion
-5. Run EFAST fusion
-6. Generate NDVI from fused outputs
 
-## Data Structure
+By default, most stages in `run.py` are **commented out** (metrics-only). Uncomment acquisition → pre-selection → preparation → fusion → post-processing for a full run.
 
-```
+### Pipeline stages
-data/
-  {site_name}/
-    {season}/
-      raw/
-        s2/          # Sentinel-2 GeoTIFFs
-        s3/          # Sentinel-3 GeoTIFFs
-        ndvi/        # NDVI from raw data
-      prepared/
-        s2/          # Prepared S2 data
-        s3/          # Prepared S3 data
-        fusion/      # EFAST fusion outputs
-        ndvi/        # NDVI from prepared/fused data
-      clouds.json    # Cloud detection results
-```
 
-### File Formats
+1. Download S2, S3, and PhenoCam
+2. Pre-selection (per-sensor NDVI screening → `raw/preselection/`)
+3. Prepare S2/S3 for each strategy (`prepared_{aggressive|nonaggressive}/` and `_itb/` variants)
+4. EFAST fusion (BtI reflectance and ItB GCC products)
+5. Post-process crops and timeseries (`processed_*_sigma{20,30}/`)
+6. Compute metrics vs PhenoCam → `metrics.json`
 
-**Sentinel-2 (raw/s2/)**: Multi-band GeoTIFF
+### Gap validation (optional)
-- Bands: B02 (blue), B03 (green), B04 (red), B8A (nir)
-- Metadata: `VIEWING_ZENITH_ANGLE` tag (degrees)
-- Filename: `{YYYYMMDD}_{increment}.geotiff`
 
-**Sentinel-3 (raw/s3/)**: Multi-band GeoTIFF
+With prepared data and EFAST installed:
-- Bands: SDR_Oa04 (blue), SDR_Oa06 (green), SDR_Oa08 (red), SDR_Oa17 (nir)
-- Filename: `{YYYYMMDD}_{increment}.geotiff`
-
-## Web Viewer
-
-Run a local HTTP server from the **webapp** directory:
 
 ```bash
-cd webapp
+python -m gap_validation.run --site innsbruck --season 2024 --lat 47.116171 --lon 11.320308
-python3 -m http.server 8000
 ```
 
-Then open `http://localhost:8000/` in your browser. Data is served via the `webapp/data` symlink.
+Writes `data/{site}/{season}/validation/gap_manifest.json`, `gap_validation_summary.json`, and masked fusion under `validation/fusion/`. See `python -m gap_validation.run --help`.
+
+## Data layout
+
+```
+data/{site_name}/{season}/
+  raw/
+    s2/                    # {YYYYMMDD}_{n}.geotiff — B02, B03, B04, B8A
+    s3/                    # {YYYYMMDD}_{n}.geotiff — Oa04, Oa06, Oa08, Oa17
+    phenocam/              # JPEGs, GCC JSON, phenology sidecar
+    preselection/          # {s2,s3}_preselection.{json,csv}
+  prepared_{strategy}/
+    s2/                    # REFL + DIST_CLOUD GeoTIFFs
+    s3/                    # composite_{YYYYMMDD}.tif
+    fusion/                # REFL_{YYYYMMDD}.tif (σ≈20)
+    fusion_sigma30/        # REFL (σ=30)
+  prepared_{strategy}_itb/
+    s2/  s3/  fusion/      # GCC products (Index-then-Blend)
+  processed_{strategy}_sigma{20,30}/
+    s2/  s3/  fusion/      # cropped {YYYYMMDD}_0.geotiff
+    gcc/  ndvi/            # timeseries.json per source
+  processed_{strategy}_itb_sigma{20,30}/
+    s2/  s3/  fusion/  gcc/
+  validation/            # gap experiment (when run)
+  metrics.json
+```
+
+Site metadata: `data/sites.geojson` (six thesis sites). `data/coweeta/` is local/legacy and not listed there.
+
+### File formats
+
+**Sentinel-2** — Multi-band GeoTIFF; bands `[blue, green, red, nir]`; `VIEWING_ZENITH_ANGLE` metadata; filename `{YYYYMMDD}_{increment}.geotiff`.
+
+**Sentinel-3** — Multi-band GeoTIFF; same band order; filename `{YYYYMMDD}_{increment}.geotiff`.
+
+**Prepared S2** — `S2A_MSIL2A_{YYYYMMDD}_REFL.tif` plus `*DIST_CLOUD.tif` (cloud-distance weights for EFAST).
+
+## Web viewer
+
+Static HTML/JS in `webapp/` — no build step. Shared GeoTIFF helpers: `webapp/common.js`. CDN: Leaflet, geotiff.js, proj4. Symlink: `webapp/data` → `../data`.
+
+Serve from the **repository root** (not `webapp/`):
+
+```bash
+python3 -m http.server 8000
+# http://localhost:8000/webapp/index.html
+```
+
+Or from the workspace root: `make serve`.
+
+| Page | Purpose | Primary data paths |
+|------|---------|-------------------|
+| `index.html` | Post-processed maps, NDVI/GCC timeseries, PhenoCam | `processed_{strategy}_sigma{n}/`, `raw/phenocam/` |
+| `preselection.html` | Cloud-screening diagnostics | `raw/preselection/{s2,s3}_preselection.json` |
+| `prepared.html` | Prepared REFL/GCC before crop | `prepared_{strategy}/`, `prepared_{strategy}_itb/` |
+| `fusion.html` | EFAST daily fusion rasters | `prepared_*/fusion/`, `fusion_sigma30/` |
+| `postprocessed.html` | Cropped processed stacks | `processed_*_sigma*/` |
+| `metrics.html` | Tabular `metrics.json` (thesis export source) | `{site}/{season}/metrics.json` under `webapp/data/` |
+| `gap_validation.html` | Withheld-S2 gap experiment | `{site}/{season}/validation/gap_validation_summary.json` |
+| `phenology.html` | TIMESAT on PhenoCam GCC | `raw/phenocam/phenocam_phenology.json` |
+
+Site/season dropdowns use `data/sites.geojson`. Map pages: **BtI | ItB**; scenarios `aggressive` / `nonaggressive`, σ 20 / 30. Keep the shared nav consistent across all eight pages. QA only — thesis tables are exported from the workspace root (`make export` or `../scripts/export_thesis_tables.py`).
+
+## Development
+
+```bash
+ruff check --fix . && ruff format .
+```
+
+Pre-commit hooks: `.pre-commit-config.yaml`.
 
 ## License
 
-This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). See the [LICENSE](LICENSE) file for details.
+GNU Affero General Public License v3.0 (AGPL-3.0). See [LICENSE](LICENSE).

fusion.py

@@ -24,14 +24,20 @@ def run_efast(
     cleaning_strategy="aggressive",
     sigma=None,
     date_range=None,
+    *,
+    s2_output_dir=None,
+    s3_output_dir=None,
+    fusion_output_dir=None,
 ):
     lat, lon = site_position
     datetime_range = date_range or f"{season}-01-01/{season}-12-31"
 
     efast_base_dir = _get_base_dir(season, site_name, cleaning_strategy)
-    s2_output_dir = efast_base_dir / "s2"
+    s2_output_dir = s2_output_dir or (efast_base_dir / "s2")
-    s3_output_dir = efast_base_dir / "s3"
+    s3_output_dir = s3_output_dir or (efast_base_dir / "s3")
-    fusion_output_dir = efast_base_dir / (f"fusion_sigma{sigma}" if sigma else "fusion")
+    fusion_output_dir = fusion_output_dir or (
+        efast_base_dir / (f"fusion_sigma{sigma}" if sigma else "fusion")
+    )
 
     fusion_output_dir.mkdir(parents=True, exist_ok=True)
     print(f"[EFAST] Starting fusion: {site_name} ({lat:.6f}, {lon:.6f}), {season}")
@@ -101,13 +107,19 @@ def run_efast_itb(
     cleaning_strategy="aggressive",
     sigma=None,
     date_range=None,
+    *,
+    s2_output_dir=None,
+    s3_output_dir=None,
+    fusion_output_dir=None,
 ):
     lat, lon = site_position
     datetime_range = date_range or f"{season}-01-01/{season}-12-31"
     efast_base_dir = _get_itb_base_dir(season, site_name, cleaning_strategy)
-    s2_output_dir = efast_base_dir / "s2"
+    s2_output_dir = s2_output_dir or (efast_base_dir / "s2")
-    s3_output_dir = efast_base_dir / "s3"
+    s3_output_dir = s3_output_dir or (efast_base_dir / "s3")
-    fusion_output_dir = efast_base_dir / (f"fusion_sigma{sigma}" if sigma else "fusion")
+    fusion_output_dir = fusion_output_dir or (
+        efast_base_dir / (f"fusion_sigma{sigma}" if sigma else "fusion")
+    )
     fusion_output_dir.mkdir(parents=True, exist_ok=True)
     print(f"[EFAST-ITB] Fusion GCC: {site_name} ({lat:.6f}, {lon:.6f}), {season}")
     efast = _import_efast()

gap_validation/__init__.py

@@ -0,0 +1 @@
+"""Synthetic gap and withheld-S2 validation (outputs under data/.../validation/)."""

gap_validation/__main__.py

@@ -0,0 +1,4 @@
+from gap_validation.run import main
+
+if __name__ == "__main__":
+    main()

gap_validation/calendar.py

@@ -0,0 +1,152 @@
+"""Gap windows and nearest S2 acquisition (manifest inputs)."""
+
+from __future__ import annotations
+
+import json
+import re
+from datetime import date, datetime, timedelta
+from pathlib import Path
+
+from phenology_timesat import phenocam_phenology_path
+
+REFL_DATE_RE = re.compile(r"S2A_MSIL2A_(\d{8})_REFL\.tif$")
+
+
+def validation_dir(site_name: str, season: int) -> Path:
+    return Path(f"data/{site_name}/{season}/validation")
+
+
+def phenology_midpoint(
+    site_name: str, season: int, phenology_path: Path | None = None
+) -> date:
+    """Pick fusion gap midpoint: green-up if in season, else green-down, else July 1."""
+    path = phenology_path or phenocam_phenology_path(site_name, season)
+    y0, y1 = date(season, 1, 1), date(season, 12, 31)
+    fallback = date(season, 7, 1)
+    if not path.is_file():
+        return fallback
+    try:
+        rec = json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError):
+        return fallback
+    up_s = rec.get("green_up_50pct_date")
+    dn_s = rec.get("green_down_50pct_date")
+
+    def _parse(s) -> date | None:
+        if not s or not isinstance(s, str):
+            return None
+        try:
+            d = datetime.strptime(s[:10], "%Y-%m-%d").date()
+        except ValueError:
+            return None
+        return d if y0 <= d <= y1 else None
+
+    up, dn = _parse(up_s), _parse(dn_s)
+    if up:
+        return up
+    if dn:
+        return dn
+    return fallback
+
+
+def centered_window(mid: date, gap_days: int, season: int) -> tuple[date, date]:
+    """[start, end] inclusive, gap_days wide, clamped to calendar year."""
+    half = gap_days // 2
+    start = mid - timedelta(days=half)
+    end = mid + timedelta(days=gap_days - 1 - half)
+    y0, y1 = date(season, 1, 1), date(season, 12, 31)
+    if start < y0:
+        end = min(y1, end + (y0 - start))
+        start = y0
+    if end > y1:
+        start = max(y0, start - (end - y1))
+        end = y1
+    return start, end
+
+
+def list_s2_refl_dates(prepared_s2: Path) -> list[tuple[date, str]]:
+    """Return sorted (acquisition_date, filename) for *REFL.tif."""
+    out: list[tuple[date, str]] = []
+    if not prepared_s2.is_dir():
+        return out
+    for p in sorted(prepared_s2.glob("*REFL.tif")):
+        m = REFL_DATE_RE.search(p.name)
+        if not m:
+            continue
+        d = datetime.strptime(m.group(1), "%Y%m%d").date()
+        out.append((d, p.name))
+    out.sort(key=lambda x: x[0])
+    return out
+
+
+def nearest_s2_acquisition(
+    prediction: date, pairs: list[tuple[date, str]]
+) -> tuple[date, str] | None:
+    if not pairs:
+        return None
+    best = min(pairs, key=lambda t: abs((t[0] - prediction).days))
+    return best
+
+
+def build_manifest_entries(
+    site_name: str,
+    season: int,
+    gap_lengths: tuple[int, ...] = (15, 30, 60, 90),
+    s2_calendar_strategy: str = "aggressive",
+) -> list[dict]:
+    """One entry per gap length: window, prediction=midpoint, withheld = nearest S2 to midpoint."""
+    mid = phenology_midpoint(site_name, season)
+    prepared_s2 = Path(f"data/{site_name}/{season}/prepared_{s2_calendar_strategy}/s2")
+    pairs = list_s2_refl_dates(prepared_s2)
+    entries = []
+    for gap_days in gap_lengths:
+        w0, w1 = centered_window(mid, gap_days, season)
+        prediction = mid
+        ns = nearest_s2_acquisition(prediction, pairs)
+        if ns is None:
+            withheld_date = None
+            withheld_filename = None
+        else:
+            withheld_date, withheld_filename = ns[0].isoformat(), ns[1]
+        entries.append(
+            {
+                "gap_days": gap_days,
+                "midpoint_rule": "green_up_50pct else green_down_50pct else July01",
+                "midpoint_date": mid.isoformat(),
+                "window_start": w0.isoformat(),
+                "window_end": w1.isoformat(),
+                "prediction_date": prediction.isoformat(),
+                "withheld_s2_date": withheld_date,
+                "withheld_s2_filename": withheld_filename,
+            }
+        )
+    return entries
+
+
+def write_manifest(
+    site_name: str,
+    season: int,
+    site_position: tuple[float, float],
+    s2_calendar_strategy: str = "aggressive",
+) -> Path:
+    out_dir = validation_dir(site_name, season)
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / "gap_manifest.json"
+    payload = {
+        "site_name": site_name,
+        "season": season,
+        "site_position_lat_lon": list(site_position),
+        "s2_calendar_strategy": s2_calendar_strategy,
+        "entries": build_manifest_entries(
+            site_name, season, s2_calendar_strategy=s2_calendar_strategy
+        ),
+    }
+    path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
+    return path
+
+
+def load_manifest(site_name: str, season: int) -> dict:
+    path = validation_dir(site_name, season) / "gap_manifest.json"
+    if not path.is_file():
+        raise FileNotFoundError(f"Missing manifest: {path}")
+    return json.loads(path.read_text(encoding="utf-8"))

gap_validation/fusion_masked.py

@@ -0,0 +1,113 @@
+"""EFAST with symlinked S2 dir (withhold one acquisition); outputs under validation/."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from tempfile import TemporaryDirectory
+
+from fusion import run_efast, run_efast_itb
+from preparation import _get_base_dir, _get_itb_base_dir
+
+from gap_validation.s2_mask_dir import build_masked_s2_dir_bti, build_masked_s2_dir_itb
+
+
+def prepared_s3_dir(season: int, site_name: str, strategy: str) -> Path:
+    return _get_base_dir(season, site_name, strategy) / "s3"
+
+
+def validation_fusion_dir(
+    site_name: str,
+    season: int,
+    gap_days: int,
+    strategy: str,
+    sigma: int | None,
+    mode: str,
+) -> Path:
+    """``data/.../validation/fusion/gap_{n}/{strategy}_sigma{20|30}_{bti|itb}/``."""
+    sig = 30 if sigma == 30 else 20
+    return (
+        Path(f"data/{site_name}/{season}/validation")
+        / "fusion"
+        / f"gap_{gap_days}"
+        / f"{strategy}_sigma{sig}_{mode}"
+    )
+
+
+def run_masked_fusion_one_date(
+    season: int,
+    site_position: tuple[float, float],
+    site_name: str,
+    strategy: str,
+    sigma: int | None,
+    mode: str,
+    prediction_date_iso: str,
+    withheld_yyyymmdd: str,
+    fusion_output_dir: Path,
+) -> Path:
+    """Build temp masked S2 dir, run EFAST for ``prediction_date_iso`` only; return output dir."""
+    fusion_output_dir.mkdir(parents=True, exist_ok=True)
+    date_range = f"{prediction_date_iso[:10]}/{prediction_date_iso[:10]}"
+    s3_dir = prepared_s3_dir(season, site_name, strategy)
+
+    with TemporaryDirectory(prefix="gapval_s2_") as tmp:
+        tmp_s2 = Path(tmp) / "s2"
+        if mode == "bti":
+            prep_s2 = _get_base_dir(season, site_name, strategy) / "s2"
+            build_masked_s2_dir_bti(prep_s2, withheld_yyyymmdd, tmp_s2)
+            run_efast(
+                season,
+                site_position,
+                site_name,
+                cleaning_strategy=strategy,
+                sigma=sigma,
+                date_range=date_range,
+                s2_output_dir=tmp_s2,
+                s3_output_dir=s3_dir,
+                fusion_output_dir=fusion_output_dir,
+            )
+        elif mode == "itb":
+            prep_s2 = _get_itb_base_dir(season, site_name, strategy) / "s2"
+            s3_itb = _get_itb_base_dir(season, site_name, strategy) / "s3"
+            build_masked_s2_dir_itb(prep_s2, withheld_yyyymmdd, tmp_s2)
+            run_efast_itb(
+                season,
+                site_position,
+                site_name,
+                cleaning_strategy=strategy,
+                sigma=sigma,
+                date_range=date_range,
+                s2_output_dir=tmp_s2,
+                s3_output_dir=s3_itb,
+                fusion_output_dir=fusion_output_dir,
+            )
+        else:
+            raise ValueError(f"mode must be bti or itb, got {mode!r}")
+
+    return fusion_output_dir
+
+
+def production_fusion_path(
+    season: int,
+    site_name: str,
+    strategy: str,
+    sigma: int | None,
+    mode: str,
+    yyyymmdd: str,
+) -> Path:
+    """Single-date fused raster from the normal prepared tree (no-gap baseline)."""
+    if mode == "bti":
+        base = _get_base_dir(season, site_name, strategy)
+        sub = f"fusion_sigma{sigma}" if sigma else "fusion"
+        return base / sub / f"REFL_{yyyymmdd}.tif"
+    base = _get_itb_base_dir(season, site_name, strategy)
+    sub = f"fusion_sigma{sigma}" if sigma else "fusion"
+    return base / sub / f"GCC_{yyyymmdd}.tif"
+
+
+def withheld_s2_refl_path(
+    season: int, site_name: str, strategy: str, withheld_filename: str | None
+) -> Path | None:
+    if not withheld_filename:
+        return None
+    p = _get_base_dir(season, site_name, strategy) / "s2" / withheld_filename
+    return p if p.is_file() else None

gap_validation/run.py

@@ -0,0 +1,290 @@
+"""Tier-2 gap validation CLI: manifest, masked EFAST, spatial ``nse_s2``, Whittaker crossover."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import subprocess
+import sys
+from datetime import datetime
+from pathlib import Path
+
+from gap_validation.calendar import load_manifest, validation_dir, write_manifest
+from gap_validation.fusion_masked import (
+    production_fusion_path,
+    run_masked_fusion_one_date,
+    validation_fusion_dir,
+    withheld_s2_refl_path,
+)
+from gap_validation.spatial_metrics import evaluate_gap_vs_withheld
+from gap_validation.whittaker_compare import first_gap_where_fusion_below_whittaker
+
+
+def _ymd_from_iso(iso_d: str) -> str:
+    return datetime.strptime(iso_d[:10], "%Y-%m-%d").strftime("%Y%m%d")
+
+
+def _yyyymmdd_from_withheld_filename(fn: str) -> str | None:
+    for part in fn.replace(".tif", "").split("_"):
+        if len(part) == 8 and part.isdigit():
+            return part
+    return None
+
+
+def _withheld_iso(entry: dict) -> str | None:
+    d = entry.get("withheld_s2_date")
+    if isinstance(d, str) and len(d) >= 10:
+        return d[:10]
+    fn = entry.get("withheld_s2_filename")
+    if not fn or not isinstance(fn, str):
+        return None
+    ymd = _yyyymmdd_from_withheld_filename(fn)
+    if not ymd:
+        return None
+    return datetime.strptime(ymd, "%Y%m%d").date().isoformat()
+
+
+def _fused_file(fusion_dir: Path, mode: str, ymd: str) -> Path:
+    stem = "REFL" if mode == "bti" else "GCC"
+    return fusion_dir / f"{stem}_{ymd}.tif"
+
+
+def _scenario_key(strategy: str, sigma: int | None, mode: str) -> str:
+    sig = 30 if sigma == 30 else 20
+    return f"{strategy}_sigma{sig}_{mode}"
+
+
+def _git_rev() -> str | None:
+    try:
+        return subprocess.check_output(
+            ["git", "rev-parse", "HEAD"],
+            cwd=Path(__file__).resolve().parent.parent,
+            text=True,
+        ).strip()
+    except (OSError, subprocess.CalledProcessError):
+        return None
+
+
+def run_validation(
+    site_name: str,
+    season: int,
+    site_position: tuple[float, float],
+    strategy: str,
+    sigma: int | None,
+    mode: str,
+    *,
+    skip_manifest: bool,
+    skip_fusion: bool,
+    write_manifest_only: bool,
+    gap_days_filter: list[int] | None,
+    s2_calendar_strategy: str,
+) -> Path:
+    base = Path(f"data/{site_name}/{season}")
+    vdir = validation_dir(site_name, season)
+    vdir.mkdir(parents=True, exist_ok=True)
+
+    if not skip_manifest:
+        write_manifest(
+            site_name, season, site_position, s2_calendar_strategy=s2_calendar_strategy
+        )
+    if write_manifest_only:
+        return vdir / "gap_manifest.json"
+
+    manifest = load_manifest(site_name, season)
+    entries = manifest["entries"]
+    if gap_days_filter:
+        entries = [e for e in entries if e.get("gap_days") in gap_days_filter]
+
+    results: list[dict] = []
+    for entry in entries:
+        gap_days = entry["gap_days"]
+        pred = entry["prediction_date"]
+        fn = entry.get("withheld_s2_filename")
+        if not fn:
+            results.append(
+                {
+                    "gap_days": gap_days,
+                    "error": "no_withheld_s2_filename",
+                    "entry": entry,
+                }
+            )
+            continue
+        ymd = _ymd_from_iso(pred)
+        wh_ymd = _yyyymmdd_from_withheld_filename(fn)
+        if not wh_ymd:
+            results.append(
+                {
+                    "gap_days": gap_days,
+                    "error": "could_not_parse_withheld_yyyymmdd",
+                    "withheld_s2_filename": fn,
+                }
+            )
+            continue
+        withheld_iso = (
+            _withheld_iso(entry) or f"{wh_ymd[:4]}-{wh_ymd[4:6]}-{wh_ymd[6:8]}"
+        )
+
+        fusion_out = validation_fusion_dir(
+            site_name, season, gap_days, strategy, sigma, mode
+        )
+        if not skip_fusion:
+            run_masked_fusion_one_date(
+                season,
+                site_position,
+                site_name,
+                strategy,
+                sigma,
+                mode,
+                pred,
+                wh_ymd,
+                fusion_out,
+            )
+
+        fused_gap = _fused_file(fusion_out, mode, ymd)
+        prod = production_fusion_path(season, site_name, strategy, sigma, mode, ymd)
+        wh_path = withheld_s2_refl_path(season, site_name, strategy, fn)
+        if wh_path is None or not fused_gap.is_file():
+            results.append(
+                {
+                    "gap_days": gap_days,
+                    "prediction_date": pred,
+                    "withheld_s2_filename": fn,
+                    "scenario": {
+                        "strategy": strategy,
+                        "sigma": 30 if sigma == 30 else 20,
+                        "mode": mode,
+                    },
+                    "error": "missing_withheld_refl_or_fused_gap",
+                    "fused_gap_path": str(fused_gap),
+                }
+            )
+            continue
+
+        spatial = evaluate_gap_vs_withheld(
+            wh_path,
+            fused_gap,
+            prod if prod.is_file() else None,
+            mode,
+            whittaker_context=(base, strategy, pred, withheld_iso),
+        )
+        fusion_nse = (spatial.get("gap") or {}).get("nse_s2")
+        wh_nse = (spatial.get("whittaker") or {}).get("nse_s2")
+        results.append(
+            {
+                "gap_days": gap_days,
+                "prediction_date": pred,
+                "withheld_s2_filename": fn,
+                "scenario": {
+                    "strategy": strategy,
+                    "sigma": 30 if sigma == 30 else 20,
+                    "mode": mode,
+                },
+                "paths": {
+                    "fused_gap": str(fused_gap),
+                    "fused_no_gap": str(prod) if prod.is_file() else None,
+                    "withheld_s2_refl": str(wh_path),
+                },
+                "spatial": spatial,
+                "whittaker_crossover_row": {
+                    "gap_days": gap_days,
+                    "nse_s2_fusion": fusion_nse,
+                    "nse_s2_whittaker": wh_nse,
+                },
+            }
+        )
+
+    scenario = _scenario_key(strategy, sigma, mode)
+    crossover_rows = [
+        r["whittaker_crossover_row"]
+        for r in results
+        if isinstance(r.get("whittaker_crossover_row"), dict)
+    ]
+    summary = {
+        "site_name": site_name,
+        "season": season,
+        "scenario": scenario,
+        "command_line": sys.argv,
+        "git_commit": _git_rev(),
+        "manifest": str(vdir / "gap_manifest.json"),
+        "results": results,
+        "whittaker_crossover": {
+            scenario: {
+                "metric": "nse_s2_spatial_vs_withheld_s2_gcc",
+                "whittaker_definition": (
+                    "Whittaker λ=400 d² on cloud-screened S2 GCC from s2_preselection.json; "
+                    "withheld acquisition removed from the fit; prediction is a spatially constant "
+                    "field at the smoothed GCC(prediction_date), compared to withheld S2 GCC on the "
+                    "same valid mask as fusion (aligned with baseline.s2_whittaker_lambda400 spirit)."
+                ),
+                "first_gap_days_fusion_nse_below_whittaker": first_gap_where_fusion_below_whittaker(
+                    crossover_rows,
+                    fusion_key="nse_s2_fusion",
+                    whittaker_key="nse_s2_whittaker",
+                ),
+                "by_gap": crossover_rows,
+            }
+        },
+    }
+    out_path = vdir / "gap_validation_summary.json"
+    out_path.write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8")
+    return out_path
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(
+        description="Tier-2 withheld-S2 gap validation (outputs under data/.../validation/)."
+    )
+    ap.add_argument("--site", required=True)
+    ap.add_argument("--season", type=int, required=True)
+    ap.add_argument("--lat", type=float, required=True)
+    ap.add_argument("--lon", type=float, required=True)
+    ap.add_argument(
+        "--strategy", default="aggressive", choices=["aggressive", "nonaggressive"]
+    )
+    ap.add_argument("--sigma", type=int, default=20, choices=[20, 30])
+    ap.add_argument("--mode", default="bti", choices=["bti", "itb"])
+    ap.add_argument(
+        "--gap-days",
+        type=int,
+        action="append",
+        metavar="N",
+        help="Restrict to gap length(s); repeatable (default: all manifest lengths).",
+    )
+    ap.add_argument("--skip-manifest", action="store_true")
+    ap.add_argument(
+        "--skip-fusion",
+        action="store_true",
+        help="Reuse existing validation fusion rasters.",
+    )
+    ap.add_argument(
+        "--write-manifest-only",
+        action="store_true",
+        help="Write gap_manifest.json and exit (no EFAST).",
+    )
+    ap.add_argument(
+        "--s2-calendar-strategy",
+        default="aggressive",
+        choices=["aggressive", "nonaggressive"],
+        help="Which prepared_*/s2 tree is used to pick nearest S2 for withholding.",
+    )
+    args = ap.parse_args()
+    sigma_kw = 30 if args.sigma == 30 else None
+    site_position = (args.lat, args.lon)
+    out = run_validation(
+        args.site,
+        args.season,
+        site_position,
+        args.strategy,
+        sigma_kw,
+        args.mode,
+        skip_manifest=args.skip_manifest,
+        skip_fusion=args.skip_fusion,
+        write_manifest_only=args.write_manifest_only,
+        gap_days_filter=args.gap_days,
+        s2_calendar_strategy=args.s2_calendar_strategy,
+    )
+    print(out)
+
+
+if __name__ == "__main__":
+    main()

gap_validation/s2_mask_dir.py

@@ -0,0 +1,51 @@
+"""Symlink prepared S2 into a temp dir, omitting one acquisition (REFL + DIST_CLOUD)."""
+
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+# Acquisition calendar day in prepared S2 names (BtI REFL/DIST; ItB GCC/DIST).
+S2_PREP_DATE_RE = re.compile(r"_(\d{8})_(?:REFL|GCC|DIST_CLOUD)\.tif$", re.IGNORECASE)
+
+
+def yyyymmdd_in_name(name: str) -> str | None:
+    m = S2_PREP_DATE_RE.search(name)
+    return m.group(1) if m else None
+
+
+def build_masked_s2_dir(
+    prepared_s2: Path, withheld_yyyymmdd: str, dest: Path, patterns: tuple[str, ...]
+) -> int:
+    """Symlink all files matching ``patterns`` except the withheld acquisition day."""
+    dest.mkdir(parents=True, exist_ok=True)
+    n = 0
+    for pattern in patterns:
+        for src in sorted(prepared_s2.glob(pattern)):
+            if not src.is_file() and not src.is_symlink():
+                continue
+            y = yyyymmdd_in_name(src.name)
+            if y == withheld_yyyymmdd:
+                continue
+            link = dest / src.name
+            if link.exists() or link.is_symlink():
+                link.unlink()
+            link.symlink_to(src.resolve())
+            n += 1
+    return n
+
+
+def build_masked_s2_dir_bti(
+    prepared_s2: Path, withheld_yyyymmdd: str, dest: Path
+) -> int:
+    return build_masked_s2_dir(
+        prepared_s2, withheld_yyyymmdd, dest, ("*REFL.tif", "*DIST_CLOUD.tif")
+    )
+
+
+def build_masked_s2_dir_itb(
+    prepared_s2: Path, withheld_yyyymmdd: str, dest: Path
+) -> int:
+    return build_masked_s2_dir(
+        prepared_s2, withheld_yyyymmdd, dest, ("*GCC.tif", "*DIST_CLOUD.tif")
+    )

gap_validation/spatial_metrics.py

@@ -0,0 +1,187 @@
+"""Per-pixel GCC vs withheld S2; NSE (nse_s2); no-gap baseline; deltas."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import numpy as np
+import rasterio
+from rasterio.warp import reproject, Resampling
+from scipy.stats import pearsonr
+
+# Match postprocessing valid mask on reflectance (METH / postprocessing.py).
+VALID_REFL_THRESHOLD = 0.001
+
+
+def _gcc_from_rgb(blue: np.ndarray, green: np.ndarray, red: np.ndarray) -> np.ndarray:
+    t = red.astype(np.float64) + green.astype(np.float64) + blue.astype(np.float64)
+    out = np.full_like(blue, np.nan, dtype=np.float64)
+    m = (
+        np.isfinite(t)
+        & (t > 0)
+        & np.isfinite(blue)
+        & np.isfinite(green)
+        & np.isfinite(red)
+    )
+    out[m] = green[m].astype(np.float64) / t[m]
+    return out.astype(np.float32)
+
+
+def read_fused_gcc(fusion_path: Path) -> tuple[np.ndarray, dict]:
+    """Fused GCC: BtI from 4-band REFL or ItB single-band GCC."""
+    with rasterio.open(fusion_path) as src:
+        if src.count >= 4:
+            b = src.read(1).astype(np.float32)
+            g = src.read(2).astype(np.float32)
+            r = src.read(3).astype(np.float32)
+            gcc = _gcc_from_rgb(b, g, r)
+        else:
+            gcc = src.read(1).astype(np.float32)
+        prof = src.profile.copy()
+    return gcc, prof
+
+
+def warp_refl_bands_to_grid(
+    refl_path: Path,
+    height: int,
+    width: int,
+    transform,
+    crs,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+    """Resample S2 REFL blue/green/red to fusion grid (bilinear)."""
+    with rasterio.open(refl_path) as src:
+        b = np.empty((height, width), dtype=np.float32)
+        g = np.empty((height, width), dtype=np.float32)
+        r = np.empty((height, width), dtype=np.float32)
+        for i, dst in enumerate((b, g, r), start=1):
+            reproject(
+                source=rasterio.band(src, i),
+                destination=dst,
+                src_transform=src.transform,
+                src_crs=src.crs,
+                dst_transform=transform,
+                dst_crs=crs,
+                resampling=Resampling.bilinear,
+            )
+    return b, g, r
+
+
+def valid_mask_fused(fusion_path: Path, mode: str) -> np.ndarray:
+    """Valid pixels: BtI uses REFL-style mask; ItB uses single-band GCC (postprocessing ItB)."""
+    with rasterio.open(fusion_path) as src:
+        if mode == "itb" or src.count < 4:
+            d = src.read(1).astype(np.float32)
+            return np.isfinite(d) & (d > VALID_REFL_THRESHOLD)
+        stacks = src.read().astype(np.float32)
+        ok = np.isfinite(stacks).all(axis=0) & (
+            np.nanmax(stacks, axis=0) > VALID_REFL_THRESHOLD
+        )
+        return ok
+
+
+def spatial_scores(
+    y_true_gcc: np.ndarray,
+    y_pred_gcc: np.ndarray,
+    mask: np.ndarray,
+) -> dict:
+    """RMSE, MAE, mean bias, Pearson r, nse_s2 (Nash–Sutcliffe vs spatial truth)."""
+    yt = y_true_gcc[mask].astype(np.float64).ravel()
+    yp = y_pred_gcc[mask].astype(np.float64).ravel()
+    n = int(yt.size)
+    if n < 2:
+        return {"n_pixels": n}
+    mean_t = float(np.mean(yt))
+    rmse = float(np.sqrt(np.mean((yt - yp) ** 2)))
+    mae = float(np.mean(np.abs(yt - yp)))
+    bias = float(np.mean(yp - yt))
+    den = float(np.sum((yt - mean_t) ** 2))
+    nse_s2 = float(1.0 - np.sum((yt - yp) ** 2) / den) if den > 0 else None
+    r = None
+    if np.std(yt) > 0 and np.std(yp) > 0:
+        r = float(pearsonr(yt, yp)[0])
+    return {
+        "n_pixels": n,
+        "rmse": rmse,
+        "mae": mae,
+        "mean_bias": bias,
+        "pearson_r": r,
+        "nse_s2": nse_s2,
+    }
+
+
+def withheld_gcc_on_fusion_grid(
+    withheld_refl_path: Path, fused_path: Path
+) -> tuple[np.ndarray, np.ndarray, dict]:
+    """``y_true`` GCC (withheld S2) and ``y_pred`` GCC from ``fused_path``, same grid."""
+    yp, prof = read_fused_gcc(fused_path)
+    h, w = yp.shape
+    b, g, r = warp_refl_bands_to_grid(
+        withheld_refl_path, h, w, prof["transform"], prof["crs"]
+    )
+    yt = _gcc_from_rgb(b, g, r)
+    return yt, yp, prof
+
+
+def common_valid_mask(
+    yt: np.ndarray,
+    y_gap: np.ndarray,
+    y_nogap: np.ndarray | None,
+    fused_gap_path: Path,
+    mode: str,
+) -> np.ndarray:
+    """Shared finite mask: truth GCC, gap/nogap preds, and fusion valid-data rules."""
+    m = (
+        valid_mask_fused(fused_gap_path, mode)
+        & np.isfinite(yt)
+        & np.isfinite(y_gap)
+        & (yt > VALID_REFL_THRESHOLD)
+        & (y_gap > VALID_REFL_THRESHOLD)
+    )
+    if y_nogap is not None:
+        m &= np.isfinite(y_nogap) & (y_nogap > VALID_REFL_THRESHOLD)
+    return m
+
+
+def evaluate_gap_vs_withheld(
+    withheld_refl_path: Path,
+    fused_gap_path: Path,
+    fused_nogap_path: Path | None,
+    mode: str,
+    *,
+    whittaker_context: tuple[Path, str, str, str] | None = None,
+) -> dict:
+    """Spatial metrics for gap and no-gap; deltas; optional Whittaker constant-field vs same mask.
+
+    ``delta_rmse`` = RMSE_gap − RMSE_no_gap; ``delta_nse`` = NSE_no_gap − NSE_gap (higher gap loss → positive delta_nse).
+    """
+    yt, y_gap, _prof = withheld_gcc_on_fusion_grid(withheld_refl_path, fused_gap_path)
+    y_nogap = None
+    if fused_nogap_path is not None and fused_nogap_path.is_file():
+        y_nogap, _ = read_fused_gcc(fused_nogap_path)
+    mask = common_valid_mask(yt, y_gap, y_nogap, fused_gap_path, mode)
+    out: dict = {"gap": spatial_scores(yt, y_gap, mask)}
+    if y_nogap is not None:
+        out["no_gap"] = spatial_scores(yt, y_nogap, mask)
+        g, ng = out["gap"], out["no_gap"]
+        if g.get("rmse") is not None and ng.get("rmse") is not None:
+            out["delta_rmse"] = float(g["rmse"] - ng["rmse"])
+        if g.get("nse_s2") is not None and ng.get("nse_s2") is not None:
+            out["delta_nse"] = float(ng["nse_s2"] - g["nse_s2"])
+    if whittaker_context is not None:
+        from gap_validation.whittaker_compare import whittaker_gcc_on_gap_masked_series
+
+        base, strategy, prediction_iso, withheld_iso = whittaker_context
+        wgcc = whittaker_gcc_on_gap_masked_series(
+            base, strategy, prediction_iso, withheld_iso
+        )
+        if wgcc is not None:
+            out["whittaker"] = constant_field_scores(yt, float(wgcc), mask)
+    return out
+
+
+def constant_field_scores(
+    y_true_gcc: np.ndarray, scalar: float, mask: np.ndarray
+) -> dict:
+    """NSE / RMSE when prediction is a spatially constant Whittaker value (same mask as fusion)."""
+    yp = np.full_like(y_true_gcc, scalar, dtype=np.float32)
+    return spatial_scores(y_true_gcc, yp, mask)

gap_validation/whittaker_compare.py

@@ -0,0 +1,64 @@
+"""Whittaker S2 GCC (λ=400 d²) as a spatial constant vs withheld S2 GCC; crossover vs fusion nse_s2."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from metrics_stats import (
+    WHITTAKER_LAMBDA_DAYS_SQ,
+    _norm_date_key,
+    _s2_gcc_series_from_preselection,
+    _whittaker_smooth_dict,
+)
+
+
+def whittaker_gcc_on_gap_masked_series(
+    base: Path,
+    strategy: str,
+    prediction_iso: str,
+    withheld_iso: str,
+    lam: float = WHITTAKER_LAMBDA_DAYS_SQ,
+) -> float | None:
+    """Whittaker smooth on cloud-screened S2 GCC **excluding** the withheld acquisition day.
+
+    Comparator aligned with ``baseline.s2_whittaker_lambda400`` in ``metrics_stats`` (same λ,
+    same preselection GCC), but the withheld date is removed so the smoother does not see
+    the target acquisition. Value at ``prediction_iso`` (YYYY-MM-DD) is returned.
+    """
+    pred_k = _norm_date_key(prediction_iso)
+    wh_k = _norm_date_key(withheld_iso)
+    if not pred_k or not wh_k:
+        return None
+    all_gcc, flags = _s2_gcc_series_from_preselection(base)
+    if not all_gcc:
+        return None
+    idx = 0 if strategy == "aggressive" else 1
+    kept = sorted(
+        (d, g)
+        for d, g in all_gcc.items()
+        if d in flags and not flags[d][idx] and _norm_date_key(d) != wh_k
+    )
+    if len(kept) < 2:
+        return None
+    obs_d, obs_v = zip(*kept)
+    smooth = _whittaker_smooth_dict(obs_d, obs_v, lam)
+    return smooth.get(pred_k)
+
+
+def first_gap_where_fusion_below_whittaker(
+    rows: list[dict],
+    *,
+    fusion_key: str = "nse_s2",
+    whittaker_key: str = "nse_s2",
+) -> int | None:
+    """Smallest ``gap_days`` where fusion[metric] < whittaker[metric] (strict)."""
+    eligible = [
+        r
+        for r in rows
+        if r.get(fusion_key) is not None and r.get(whittaker_key) is not None
+    ]
+    eligible.sort(key=lambda r: r["gap_days"])
+    for r in eligible:
+        if r[fusion_key] < r[whittaker_key]:
+            return int(r["gap_days"])
+    return None

metrics_stats.py

@@ -64,7 +64,7 @@ def pearson_correlation(y_true, y_pred):
 
 
 def r_squared(y_true, y_pred):
-    """Calculate coefficient of determination R²."""
+    """Generalized R² vs predicting mean(y_true); can be negative. Same formula as ``nse`` with the same arguments; not Pearson r squared."""
     if len(y_true) < 2 or np.std(y_true) == 0:
         return None
     ss_res = np.sum((y_true - y_pred) ** 2)

webapp/fusion.html

@@ -44,6 +44,7 @@
                 <a href="fusion.html" class="active">Fusion</a>
                 <a href="postprocessed.html">Postprocessed</a>
                 <a href="metrics.html">Metrics</a>
+                <a href="gap_validation.html">Gap validation</a>
                 <a href="phenology.html">Phenology</a>
             </div>
             <h1 id="siteName">Innsbruck</h1>

webapp/gap_validation.html

@@ -0,0 +1,283 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <meta charset="utf-8">
+    <title>Gap validation</title>
+    <style>
+        body { margin: 0; font-family: sans-serif; }
+        .nav { margin-bottom: 15px; font-size: 14px; }
+        .nav a { margin-right: 12px; color: #0066cc; text-decoration: none; }
+        .nav a:hover { text-decoration: underline; }
+        .nav a.active { font-weight: bold; }
+        .container { max-width: 1100px; margin: 0 auto; padding: 20px; }
+        .selectors { margin-bottom: 18px; }
+        .selectors select { padding: 5px 10px; font-size: 14px; margin-right: 15px; }
+        h1 { font-size: 22px; margin-top: 0; }
+        h2 { font-size: 16px; margin-top: 22px; color: #333; }
+        h2:first-of-type { margin-top: 8px; }
+        table { border-collapse: collapse; width: 100%; font-size: 12px; margin-bottom: 14px; }
+        th, td { border: 1px solid #ccc; padding: 6px 8px; text-align: left; vertical-align: top; }
+        th { background: #f5f5f5; }
+        td.num { text-align: right; font-variant-numeric: tabular-nums; }
+        td.paths { font-size: 11px; word-break: break-all; color: #444; max-width: 420px; }
+        .intro { font-size: 13px; color: #333; background: #fafafa; border: 1px solid #e5e5e5;
+                 padding: 10px 12px; border-radius: 4px; margin-bottom: 16px; line-height: 1.5; }
+        .intro code { background: #f1f1f1; padding: 1px 4px; border-radius: 3px; font-size: 11px; }
+        .section-note { font-size: 12px; color: #555; margin: -6px 0 8px 0; line-height: 1.45; }
+        .empty { color: #666; font-style: italic; }
+        .err { color: #a00; }
+        details.meta { font-size: 12px; margin-top: 12px; border: 1px solid #e5e5e5; border-radius: 4px; padding: 8px 12px; background: #fafafa; }
+        details.meta summary { cursor: pointer; font-weight: 600; }
+        details.meta pre { margin: 8px 0 0; overflow: auto; font-size: 11px; max-height: 200px; }
+    </style>
+</head>
+<body>
+    <div class="container">
+        <div class="nav">
+            <a href="index.html">Full</a>
+            <a href="preselection.html">Pre-selection</a>
+            <a href="prepared.html">Prepared</a>
+            <a href="fusion.html">Fusion</a>
+            <a href="postprocessed.html">Postprocessed</a>
+            <a href="metrics.html">Metrics</a>
+            <a href="gap_validation.html" class="active">Gap validation</a>
+            <a href="phenology.html">Phenology</a>
+        </div>
+        <h1 id="pageTitle">Gap validation</h1>
+        <div class="selectors">
+            <label>Site:</label>
+            <select id="siteSelect"></select>
+            <label>Season:</label>
+            <select id="seasonSelect"></select>
+        </div>
+        <div id="content"></div>
+    </div>
+    <script>
+        let siteName = "innsbruck",
+            season = "2024";
+        let availableSiteSeasons = {};
+        const urlParams = new URLSearchParams(location.search);
+
+        async function probeSummary(sn, s) {
+            try {
+                const res = await fetch(`data/${sn}/${s}/validation/gap_validation_summary.json`, {
+                    method: "HEAD",
+                });
+                return res.ok;
+            } catch {
+                return false;
+            }
+        }
+
+        function fmt(v, d = 4) {
+            if (v == null || typeof v !== "number" || !Number.isFinite(v)) return "—";
+            return v.toFixed(d);
+        }
+
+        function fmtInt(v) {
+            if (v == null || typeof v !== "number" || !Number.isFinite(v)) return "—";
+            return String(Math.round(v));
+        }
+
+        function crossoverBlock(summary) {
+            const scen = summary.scenario;
+            const wcRoot = summary.whittaker_crossover || {};
+            const wc = (scen && wcRoot[scen]) || Object.values(wcRoot)[0];
+            if (!wc) return "";
+            const first = wc.first_gap_days_fusion_nse_below_whittaker;
+            const def = wc.whittaker_definition || "";
+            let h = `<h2>Whittaker crossover (NSE<sub>S2</sub>)</h2>`;
+            h += `<p class="section-note">${def}</p>`;
+            h += `<p class="section-note"><b>First gap length (days)</b> where fusion NSE<sub>S2</sub> &lt; Whittaker NSE<sub>S2</sub> (strict): <b>${first != null ? first : "—"}</b> (none if fusion never falls below).</p>`;
+            const rows = wc.by_gap || [];
+            if (rows.length) {
+                h += `<table><tr><th>Gap days</th><th class="num">NSE<sub>S2</sub> fusion</th><th class="num">NSE<sub>S2</sub> Whittaker</th></tr>`;
+                for (const r of rows) {
+                    h += `<tr><td>${r.gap_days}</td><td class="num">${fmt(r.nse_s2_fusion, 3)}</td><td class="num">${fmt(r.nse_s2_whittaker, 3)}</td></tr>`;
+                }
+                h += `</table>`;
+            }
+            return h;
+        }
+
+        function manifestTable(manifest) {
+            if (!manifest?.entries?.length) return "";
+            let h = `<h2>Gap manifest</h2>`;
+            h += `<p class="section-note">From <code>data/${siteName}/${season}/validation/gap_manifest.json</code>. Midpoint rule: ${manifest.entries[0]?.midpoint_rule || "—"}.</p>`;
+            h += `<table><tr><th>Gap days</th><th>Prediction</th><th>Window</th><th>Withheld S2</th></tr>`;
+            for (const e of manifest.entries) {
+                const w = `${e.window_start} → ${e.window_end}`;
+                h += `<tr><td>${e.gap_days}</td><td>${e.prediction_date}</td><td>${w}</td><td>${e.withheld_s2_filename || "—"}</td></tr>`;
+            }
+            h += `</table>`;
+            return h;
+        }
+
+        function resultsTable(results) {
+            if (!results?.length) return `<p class="empty">No result rows in summary.</p>`;
+            const head = `<tr>
+                <th>Gap</th><th>Prediction</th><th>Withheld REFL</th>
+                <th class="num">RMSE<br><span style="font-weight:normal">gap</span></th>
+                <th class="num">NSE<sub>S2</sub><br><span style="font-weight:normal">gap</span></th>
+                <th class="num">RMSE<br><span style="font-weight:normal">no gap</span></th>
+                <th class="num">NSE<sub>S2</sub><br><span style="font-weight:normal">no gap</span></th>
+                <th class="num">ΔRMSE</th><th class="num">ΔNSE</th>
+                <th class="num">NSE<sub>S2</sub><br><span style="font-weight:normal">Whitt.</span></th>
+                <th class="num">n</th>
+                <th>Paths / error</th>
+            </tr>`;
+            const parts = [head];
+            for (const r of results) {
+                if (r.error) {
+                    parts.push(
+                        `<tr><td>${r.gap_days ?? "—"}</td><td colspan="10" class="err">${r.error}</td><td class="paths">${r.fused_gap_path || ""}</td></tr>`
+                    );
+                    continue;
+                }
+                const g = r.spatial?.gap || {};
+                const ng = r.spatial?.no_gap || {};
+                const wh = r.spatial?.whittaker || {};
+                const dRm = r.spatial?.delta_rmse;
+                const dNs = r.spatial?.delta_nse;
+                const p = r.paths || {};
+                const pathNote = [p.fused_gap, p.fused_no_gap, p.withheld_s2_refl].filter(Boolean).join("<br>");
+                parts.push(`<tr>
+                    <td>${r.gap_days}</td>
+                    <td>${r.prediction_date || "—"}</td>
+                    <td style="font-size:11px">${r.withheld_s2_filename || "—"}</td>
+                    <td class="num">${fmt(g.rmse)}</td>
+                    <td class="num">${fmt(g.nse_s2, 3)}</td>
+                    <td class="num">${fmt(ng.rmse)}</td>
+                    <td class="num">${fmt(ng.nse_s2, 3)}</td>
+                    <td class="num">${fmt(dRm)}</td>
+                    <td class="num">${fmt(dNs, 3)}</td>
+                    <td class="num">${fmt(wh.nse_s2, 3)}</td>
+                    <td class="num">${fmtInt(g.n_pixels)}</td>
+                    <td class="paths">${pathNote}</td>
+                </tr>`);
+            }
+            return `<table>${parts.join("")}</table>`;
+        }
+
+        function metaDetails(summary) {
+            const cmd = summary.command_line;
+            const git = summary.git_commit;
+            if (!cmd && !git) return "";
+            let h = `<details class="meta"><summary>Run metadata</summary>`;
+            if (git) h += `<p>Git: <code>${git}</code></p>`;
+            if (cmd?.length) h += `<pre>${cmd.map((x) => String(x)).join(" ")}</pre>`;
+            h += `</details>`;
+            return h;
+        }
+
+        async function render(summary, manifest) {
+            const el = document.getElementById("content");
+            if (!summary) {
+                el.innerHTML = `<p class="err">Could not load <code>data/${siteName}/${season}/validation/gap_validation_summary.json</code>.</p>
+                    <p class="section-note">From <code>processing/</code>: <code>python -m gap_validation.run --site ${siteName} --season ${season} --lat LAT --lon LON</code> (see <code>--help</code>). Serve from <code>processing/</code>: <code>python3 -m http.server 8000</code> → <code>/webapp/gap_validation.html</code> (<code>webapp/data</code> → <code>../data</code>).</p>`;
+                if (manifest?.entries) el.innerHTML += manifestTable(manifest);
+                return;
+            }
+            const scen = summary.scenario || "—";
+            const sn = summary.site_name ?? siteName;
+            const se = summary.season ?? season;
+            let html = `<div class="intro">
+                Tier-2 withheld S2, spatial GCC vs withheld scene, NSE<sub>S2</sub>, and Whittaker comparison.
+                Summary: <code>data/${sn}/${se}/validation/gap_validation_summary.json</code>.
+                Scenario in this file: <b>${scen}</b> (one run overwrites; re-run CLI for other strategy/σ/mode).
+            </div>`;
+            html += `<h2>Spatial metrics (per gap length)</h2>`;
+            html += `<p class="section-note">Reference = GCC from withheld S2 REFL (bilinear to fusion grid). Prediction = fused GCC. ΔRMSE = RMSE<sub>gap</sub> − RMSE<sub>no gap</sub>; ΔNSE = NSE<sub>no gap</sub> − NSE<sub>gap</sub>.</p>`;
+            html += resultsTable(summary.results);
+            html += crossoverBlock(summary);
+            html += metaDetails(summary);
+            if (manifest?.entries) html += manifestTable(manifest);
+            el.innerHTML = html;
+        }
+
+        async function load() {
+            let summary = null,
+                manifest = null;
+            try {
+                const r1 = await fetch(`data/${siteName}/${season}/validation/gap_validation_summary.json`);
+                summary = r1.ok ? await r1.json() : null;
+            } catch {
+                summary = null;
+            }
+            try {
+                const r2 = await fetch(`data/${siteName}/${season}/validation/gap_manifest.json`);
+                manifest = r2.ok ? await r2.json() : null;
+            } catch {
+                manifest = null;
+            }
+            await render(summary, manifest);
+            const site = window.sitesData?.features?.find((f) => f.properties?.sitename === siteName);
+            document.getElementById("pageTitle").textContent =
+                (site?.properties?.description || siteName) + " — gap validation — " + season;
+            urlParams.set("site", siteName);
+            urlParams.set("season", season);
+            history.replaceState({}, "", `?${urlParams}`);
+        }
+
+        async function init() {
+            try {
+                const res = await fetch("data/sites.geojson");
+                window.sitesData = res.ok ? await res.json() : { features: [] };
+            } catch {
+                window.sitesData = { features: [] };
+            }
+            const features = window.sitesData.features || [];
+            for (const f of features) {
+                const sn = f.properties?.sitename;
+                if (!sn) continue;
+                const seasonsFromGeo = f.properties?.seasons ? Object.keys(f.properties.seasons).sort() : [];
+                const withData = [];
+                for (const s of seasonsFromGeo) {
+                    if (await probeSummary(sn, s)) withData.push(s);
+                }
+                if (withData.length) availableSiteSeasons[sn] = withData;
+            }
+            const availableSites = Object.keys(availableSiteSeasons);
+            const siteSelect = document.getElementById("siteSelect");
+            siteSelect.innerHTML = "";
+            (availableSites.length ? availableSites.sort() : ["innsbruck"]).forEach((sn) => {
+                const opt = document.createElement("option");
+                opt.value = sn;
+                opt.textContent = sn;
+                siteSelect.appendChild(opt);
+                if (!availableSiteSeasons[sn]) availableSiteSeasons[sn] = ["2024"];
+            });
+            const urlSite = urlParams.get("site");
+            const urlSeason = urlParams.get("season");
+            const initialSite = urlSite && availableSiteSeasons[urlSite] ? urlSite : availableSites[0] || "innsbruck";
+            const initialSeason =
+                urlSeason && (availableSiteSeasons[initialSite] || []).includes(urlSeason)
+                    ? urlSeason
+                    : (availableSiteSeasons[initialSite] || [])[0] || "2024";
+            siteSelect.value = initialSite;
+            document.getElementById("seasonSelect").innerHTML = (availableSiteSeasons[initialSite] || [])
+                .map((s) => `<option value="${s}">${s}</option>`)
+                .join("");
+            document.getElementById("seasonSelect").value = initialSeason;
+            siteName = initialSite;
+            season = initialSeason;
+
+            siteSelect.addEventListener("change", function () {
+                const sn = this.value;
+                const seas = availableSiteSeasons[sn] || [];
+                document.getElementById("seasonSelect").innerHTML = seas.map((s) => `<option value="${s}">${s}</option>`).join("");
+                document.getElementById("seasonSelect").value = seas[0] || "2024";
+                siteName = sn;
+                season = document.getElementById("seasonSelect").value;
+                load();
+            });
+            document.getElementById("seasonSelect").addEventListener("change", function () {
+                season = this.value;
+                load();
+            });
+            await load();
+        }
+        init();
+    </script>
+</body>
+</html>

webapp/index.html

@@ -55,6 +55,7 @@
             <a href="fusion.html">Fusion</a>
             <a href="postprocessed.html">Postprocessed</a>
             <a href="metrics.html">Metrics</a>
+            <a href="gap_validation.html">Gap validation</a>
             <a href="phenology.html">Phenology</a>
         </div>
         <div class="slider-container">
@@ -136,7 +137,7 @@
         <div class="combined-plot">
             <div class="combined-plot-label">Metrics vs PhenoCam (fusion scenarios)</div>
             <p style="margin:4px 0 8px; font-size:11px; color:#555; max-width:720px;">
-                R² (variance explained), nRMSE (RMSE normalised by PhenoCam σ), NSE_PC (Nash–Sutcliffe vs PhenoCam). <b>ΔNSE_PC</b> = NSE_PC(σ20)−NSE_PC(σ30): positive → σ20 better; negative → σ30 better. <b>Mean residual</b> (fused−PhenoCam): positive → fusion high vs PhenoCam on average; negative → low; compare BtI vs ItB in the same row (closer to 0 = less mean bias). Tables at the top when <code>metrics.json</code> has <code>derived</code> (regenerate with <code>metrics_stats.py</code> / <code>run.py</code>).
+                <b>R² vs mean</b> (JSON <code>r_squared</code>): generalized R² vs predicting mean PhenoCam each day — same numeric value as <b>NSE_PC</b>, not (Pearson <i>r</i>)²; can be negative. <b>nRMSE</b> (RMSE / mean PhenoCam). <b>ΔNSE_PC</b> = NSE_PC(σ20)−NSE_PC(σ30): positive → σ20 better; negative → σ30 better. <b>Mean residual</b> (fused−PhenoCam): positive → fusion high vs PhenoCam on average; negative → low; compare BtI vs ItB in the same row (closer to 0 = less mean bias). Tables at the top when <code>metrics.json</code> has <code>derived</code> (regenerate with <code>metrics_stats.py</code> / <code>run.py</code>).
             </p>
             <div id="metricsTable" style="overflow-x: auto; margin-top: 10px;"></div>
         </div>
@@ -642,7 +643,7 @@
                 html +=
                     "<p style='margin:0 0 2px; font-size:11px; font-weight:600;'>Mean residual (fused − PhenoCam): BtI vs ItB</p>";
                 html +=
-                    "<p style='margin:0 0 6px; font-size:10px; color:#555; line-height:1.35;'>Each cell: mean(fused−PhenoCam) on matched dates. <b>+</b> overestimates, <b>−</b> underestimates; <b>~0</b> little mean bias (see R²/MAE for overall fit). Same row: column closer to 0 → less systematic offset vs PhenoCam (RQ1.1).</p>";
+                    "<p style='margin:0 0 6px; font-size:10px; color:#555; line-height:1.35;'>Each cell: mean(fused−PhenoCam) on matched dates. <b>+</b> overestimates, <b>−</b> underestimates; <b>~0</b> little mean bias (see R² vs mean / nRMSE / NSE_PC for overall fit). Same row: column closer to 0 → less systematic offset vs PhenoCam (RQ1.1).</p>";
                 html +=
                     "<table style='width:100%; border-collapse:collapse; font-size:11px; margin-bottom:10px;'><thead><tr style='background:#f5f5f5;'><th style='padding:6px;'>Strategy</th><th style='padding:6px;'>σ</th><th style='padding:6px; text-align:right;'>BtI</th><th style='padding:6px; text-align:right;'>ItB</th></tr></thead><tbody>";
                 for (const row of d.bti_vs_itb_mean_residual) {
@@ -654,7 +655,7 @@
             const scenarios = ["aggressive_sigma20", "aggressive_sigma30", "nonaggressive_sigma20", "nonaggressive_sigma30"];
             const scenarioNames = ["Aggressive σ20", "Aggressive σ30", "Non-aggressive σ20", "Non-aggressive σ30"];
             const metrics = ["r_squared", "nrmse", "nse_pc"];
-            const metricLabels = { r_squared: "R²", nrmse: "nRMSE", nse_pc: "NSE_PC" };
+            const metricLabels = { r_squared: "R² vs mean", nrmse: "nRMSE", nse_pc: "NSE_PC" };
 
             html += "<table style='width:100%; border-collapse:collapse; font-size:11px;'>";
             html += "<thead><tr style='background:#f5f5f5; border-bottom:2px solid #ccc;'>";

webapp/metrics.html

@@ -55,6 +55,7 @@
             <a href="fusion.html">Fusion</a>
             <a href="postprocessed.html">Postprocessed</a>
             <a href="metrics.html" class="active">Metrics</a>
+            <a href="gap_validation.html">Gap validation</a>
             <a href="phenology.html">Phenology</a>
         </div>
         <h1 id="siteName">Metrics</h1>
@@ -70,7 +71,7 @@
         /** Shown in the UI; pearson_r, rmse, mae, n_samples remain in metrics.json only. */
         const DISPLAY_METRIC_COLS = ["r_squared", "nrmse", "nse_pc"];
         const DISPLAY_METRIC_LABELS = {
-            r_squared: "R²",
+            r_squared: "R² vs mean",
             nrmse: "nRMSE",
             nse_pc: "NSE_PC",
         };
@@ -229,9 +230,9 @@
                 <summary>How to read</summary>
                 <ol>
                     <li>All scores are satellite or fusion <b>GCC</b> vs <b>PhenoCam GCC</b> at the site 3×3 window, <b>same calendar days</b> only. Extra stats: <code>metrics.json</code>.</li>
-                    <li><b>R²</b>, <b>NSE_PC</b>: higher = better. <b>nRMSE</b>: lower = better.</li>
+                    <li><b>R² vs mean</b> and <b>NSE_PC</b> are the same value (1 − SS<sub>res</sub>/SS<sub>tot</sub> vs predicting mean PhenoCam each day); not (Pearson <i>r</i>)²; can be negative. Higher = better. <b>nRMSE</b>: lower = better.</li>
                     <li><b>Fusion:</b> same row number in BtI and in ItB = same screening + same σ — compare left/right. Down one block = change screening or σ.</li>
-                    <li><b>Mean resid.</b> (if present): mean(fused − PhenoCam). Sign = average bias; use R² / nRMSE / NSE_PC for overall fit.</li>
+                    <li><b>Mean resid.</b> (if present): mean(fused − PhenoCam). Sign = average bias; use R² vs mean / nRMSE / NSE_PC for overall fit.</li>
                     <li><b>Summaries:</b> ΔNSE_PC = NSE at σ20 minus NSE at σ30 (+ means σ20 wins). Paired table: closer to 0 = less mean bias.</li>
                 </ol>
             </details>`;
@@ -245,6 +246,7 @@
                     <li><b>ItB</b>: GCC on S2 and S3, then fuse GCC.</li>
                     <li><b>Scenario</b>: screening (<code>aggressive</code> / <code>nonaggressive</code>) × σ (20 / 30 days).</li>
                     <li><a href="phenology.html">Phenology</a> — PhenoCam SOS/EOS (TIMESAT).</li>
+                    <li><b>R² vs mean</b> — coefficient of determination vs a constant mean(PhenoCam) baseline; JSON key <code>r_squared</code>; duplicates <code>nse_pc</code>. Not (Pearson <i>r</i>)².</li>
                     <li><code>metrics.json</code> — also Pearson <i>r</i>, RMSE, MAE, <code>n_samples</code>.</li>
                 </ul>
             </details>`;
@@ -274,7 +276,7 @@
             const baselineTbl = baselineTable(data.baseline);
             if (baselineTbl) {
                 html += `<h2>Baselines (vs PhenoCam)</h2>`;
-                html += `<p class="section-note">Same columns as fusion (vs PhenoCam). Higher R² / NSE_PC, lower nRMSE = better. S3 = coarse-only; Whittaker = smoothed S2-only.</p>`;
+                html += `<p class="section-note">Same columns as fusion (vs PhenoCam). Higher R² vs mean / NSE_PC, lower nRMSE = better. S3 = coarse-only; Whittaker = smoothed S2-only.</p>`;
                 html += baselineTbl;
             }
 

webapp/phenology.html

@@ -34,6 +34,7 @@
             <a href="fusion.html">Fusion</a>
             <a href="postprocessed.html">Postprocessed</a>
             <a href="metrics.html">Metrics</a>
+            <a href="gap_validation.html">Gap validation</a>
             <a href="phenology.html" class="active">Phenology</a>
         </div>
         <h1>PhenoCam phenology (50% amplitude)</h1>

webapp/postprocessed.html

@@ -44,6 +44,7 @@
                 <a href="fusion.html">Fusion</a>
                 <a href="postprocessed.html" class="active">Postprocessed</a>
                 <a href="metrics.html">Metrics</a>
+                <a href="gap_validation.html">Gap validation</a>
                 <a href="phenology.html">Phenology</a>
             </div>
             <h1 id="siteName">Innsbruck</h1>

webapp/prepared.html

@@ -44,6 +44,7 @@
                 <a href="fusion.html">Fusion</a>
                 <a href="postprocessed.html">Postprocessed</a>
                 <a href="metrics.html">Metrics</a>
+                <a href="gap_validation.html">Gap validation</a>
                 <a href="phenology.html">Phenology</a>
             </div>
             <h1 id="siteName">Innsbruck</h1>

webapp/preselection.html

@@ -43,6 +43,7 @@
                 <a href="fusion.html">Fusion</a>
                 <a href="postprocessed.html">Postprocessed</a>
                 <a href="metrics.html">Metrics</a>
+                <a href="gap_validation.html">Gap validation</a>
                 <a href="phenology.html">Phenology</a>
             </div>
             <h1 id="siteName">Innsbruck</h1>