blakeridgway/heloha
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase 0-2 complete

Browse Source
This commit is contained in:
Blake Ridgway
2026-04-25 20:31:55 -05:00
parent c1d8b17147
commit 778c52fc69
8 changed files with 1447 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files

198
TODO.md
View File

@@ -1,169 +1,103 @@
# Project Heloha
A detailed, actionable task list for building a unified, multi-radar severe weather analysis and alerting platform for Oklahoma.
A unified, multi-radar severe weather analysis and alerting platform for Oklahoma.
---
## Phase 0: Foundation & Setup (The First Week)
## Phase 0: Foundation & Setup ✅
*Goal: Establish a clean, professional Go project environment.*
- [x] **Initialize Git Repository:**
- [x] `git init`
- [x] Create a `.gitignore` file for Go and common OS files.
- [x] Create a `README.md` with the project's mission statement.
- [x] **Establish Go Project Structure:**
- [x] `go mod init github.com/blakeridgway/heloha`
- [x] Create a standard Go project layout:
- `/cmd/heloha-server`: Main application entry point.
- `/internal/radar`: Code for fetching, parsing, and processing radar data.
- `/internal/server`: HTTP handlers and server logic.
- `/internal/config`: Configuration management.
- `/web/templates`: HTML templates for the frontend.
- `/web/static`: CSS and JavaScript assets.
- [x] **Create a Basic Web Server:**
- [x] In `/cmd/heloha-server/main.go`, set up an `http.Server`.
- [x] Choose and implement a router (e.g., `go-chi/chi` is a good choice for middleware and flexibility).
- [x] Create a simple `/healthz` endpoint that returns `200 OK`.
- [x] **Set Up Build Automation:**
- [x] Create a `Makefile` with targets for:
- `build`: `go build -o bin/heloha-server ./cmd/heloha-server`
- `run`: `go run ./cmd/heloha-server`
- `test`: `go test ./...`
- `tidy`: `go mod tidy`
- [x] Git repo, `.gitignore`, `README.md`
- [x] Go module (`github.com/blakeridgway/heloha`), standard project layout
- [x] `go-chi/chi` HTTP router with middleware
- [x] `/healthz` endpoint
- [x] `Makefile` with `build`, `run`, `test`, `tidy`
---
## Phase 1: Single Radar Ingestion & Display (The MVP)
## Phase 1: Single Radar Ingestion & Display ✅
*Goal: Get raw data from one radar onto a map on a webpage. This proves the core data pipeline is viable.*
*Approach changed from AWS S3 → NWS public FTP (`tgftp.nws.noaa.gov`). No credentials required.*
- [ ] **Data Acquisition (`/internal/radar/fetch.go`):**
- [ ] Add the AWS SDK for Go V2 (`aws-sdk-go-v2`) as a dependency.
- [ ] **Use NEXRAD Level 3 data** from the `unidata-nexrad-level3` S3 bucket. Level 3 products are pre-processed by the RPG into a simple binary format — no polar-to-Cartesian projection needed, and no per-record bzip2 complexity.
- [ ] Implement logic to find the *latest* file for a given radar (`KTLX`) and product code (`N0B` = Base Reflectivity 0.5° tilt) using the S3 key naming convention.
- [ ] Write a function to download a specific Level 3 file from S3 into memory.
- [ ] *Note: Level 2 data (raw I/Q + full velocity fields) will be introduced in Phase 3 when TVS detection requires it. Level 3 is sufficient for reflectivity display and mosaic.*
- [ ] **Data Parsing (`/internal/radar/parse.go`):**
- [ ] Implement a Level 3 product parser targeting the Graphic Product Message format (ICD 2620001). The header is fixed-width; the symbology block contains pre-gridded radial data.
- [ ] Define Go `structs` to hold the parsed data: a `RadarProduct` struct with base reflectivity (a 2D array of floats), timestamp, elevation angle, and radar site coordinates.
- [ ] **Visualization Backend (`/internal/server/handlers.go`):**
- [ ] Create a new HTTP handler for map tiles: `/api/v1/tile/ktlx/{z}/{x}/{y}.png`.
- [ ] Inside the handler, implement logic to:
- Load the latest parsed `KTLX` data.
- Use the Z/X/Y tile coordinates to determine the required geographic bounds.
- Map the pre-gridded radar data onto the requested tile canvas (no polar projection needed at this stage).
- Use a color lookup table (LUT) to convert dBZ values to colors (e.g., green for light rain, red for heavy).
- Render the final tile as a PNG image using Go's standard `image` package.
- [ ] **Tile Caching (`/internal/server/tilecache.go`):**
- [ ] Implement an in-memory tile cache as a `map[string][]byte` (key: `"z/x/y"`) protected by a `sync.RWMutex`.
- [ ] Wrap the tile handler: check the cache first; on a miss, render the PNG, store it in the cache, and return it.
- [ ] Tie cache invalidation to the data update cycle — when a new `RadarProduct` is ingested, clear the cache for that radar. This ensures tiles are never stale beyond one update interval.
- [ ] **Frontend Display (`/web/templates/index.html`):**
- [ ] Set up a basic HTML page that includes Leaflet.js from a CDN.
- [ ] Initialize a Leaflet map centered on Oklahoma.
- [ ] Add a `L.tileLayer` pointing to your new `/api/v1/tile/ktlx/{z}/{x}/{y}.png` endpoint.
- [ ] Add the HTMX script tag.
- [ ] Use `hx-get` and `hx-trigger="every 60s"` on a container element to periodically refresh the map layer or associated metadata.
- [x] Fetch latest Level 3 N0Q (Digital Base Reflectivity 0.5°) via HTTP from NWS FTP
- [x] Parse NEXRAD Level 3 ICD-2620001 binary (WMO header strip, bzip2 Symbology Block, Packet 16 radials)
- [x] Web Mercator tile renderer with bilinear interpolation between radials and range bins
- [x] In-memory tile cache (`sync.RWMutex` map), invalidated on each ingest cycle
- [x] Leaflet.js frontend centered on Oklahoma, dark CartoDB basemap
- [x] HUD with site label, scan time, UTC + CDT live clock
- [x] dBZ color legend
- [x] 20 dBZ minimum threshold + 4-pass speckle filter to suppress AP/biological noise
---
## Phase 2: Multi-Radar Fusion (The Core Challenge)
## Phase 2: Multi-Radar Fusion ✅
*Goal: Transition from a single, siloed view to a single, authoritative statewide weather picture.*
*Nearest-radar compositing done per-pixel at tile render time — no pre-built mosaic grid needed.*
- [ ] **Concurrent Ingestion (`/internal/radar/manager.go`):**
- [ ] Design a concurrent manager that spawns one "fetcher" goroutine per radar (KTLX, KINX, KFDR, KVNX, etc.).
- [ ] Each fetcher goroutine periodically checks for new data for its assigned radar.
- [ ] Use a buffered Go `channel` to pass newly downloaded and parsed `RadarProduct` objects to a central processing component.
- [ ] **Data Mosaicing (`/internal/radar/mosaic.go`):**
- [ ] **Grid Definition:** Define a constant statewide grid in your code. This includes its geographic boundaries (min/max lat/lon), resolution (e.g., 1km x 1km), and dimensions (width/height in pixels).
- [ ] **Projection Logic:** Implement robust math functions to convert from (radar, azimuth, range) -> (lat, lon) -> (grid X, grid Y). This is crucial.
- [ ] **Compositing Engine:**
- Create a function that runs in a loop, triggered by new data or a timer.
- This function initializes a new, empty statewide grid.
- It iterates through every cell of the grid. For each cell, it checks all available radars to see which one provides the best data (lowest beam altitude above that grid point).
- It then "paints" the data from the best radar onto that grid cell.
- [ ] **State Management:** Store the latest completed mosaic grid in memory, protected by a mutex, so it can be safely accessed by the tile-serving handler.
- [ ] **Update Visualization Backend:**
- [ ] Modify the `/api/v1/tile/...` handler to render tiles from the statewide mosaic grid instead of a single radar product.
- [ ] The tile handler no longer needs the radar ID in the URL. New URL: `/api/v1/tile/reflectivity/{z}/{x}/{y}.png`.
- [x] 15 NEXRAD sites (OK, TX, KS, AR, MO, LA) ingested concurrently every 2 minutes
- [x] Per-site, per-product ring buffer (12 frames of history)
- [x] Composite tile endpoint: nearest radar per pixel within 230 km range
- [x] NWS active warnings overlay (GeoJSON from `api.weather.gov`, auto-refreshes every 2 min)
- [x] Range rings (100 / 200 km) and site markers with tooltips on all 15 sites
- [x] Loop animation: scrubber + play/pause, 500 ms/frame, builds up to 12 frames (~24 min)
- [x] REFL / VEL product toggle (velocity infrastructure built; NWS FTP returns 403 for p99r0)
- [x] Leaflet scale bar
---
## Phase 3: Advanced Analysis & Alerting
*Goal: Find the "so what?" in the data. Identify and track dangerous storms.*
*Goal: Find the "so what?" — identify, track, and score dangerous storms.*
- [ ] **Expand Data Parsing (introduce Level 2):**
- [ ] Add Level 2 ingestion from the `noaa-nexrad-level2` S3 bucket alongside the existing Level 3 pipeline. Level 2 files are compressed with bzip2 on a per-record basis — implement record-level decompression.
- [ ] Parse Level 2 Velocity, Differential Reflectivity (ZDR), and Correlation Coefficient (CC) data fields. The official format spec is ICD RDA/RPG 2620002U.
- [ ] Update the mosaicing engine to create composite grids for these new data types.
- [ ] **Storm Cell Identification (`/internal/analysis/segmentation.go`):**
- [ ] Implement a "blob detection" algorithm on the composite reflectivity grid. This can be a simple algorithm like flood-fill or a more advanced one like DBSCAN.
- [ ] The output should be a list of distinct `StormCell` objects, each with a polygon defining its boundary.
- [ ] **Signature Detection (`/internal/analysis/signatures.go`):**
- [ ] For each identified `StormCell`, analyze the underlying data within its polygon.
- [ ] Implement a TVS (Tornadic Vortex Signature) detector: Search for strong, compact, inbound/outbound velocity couplets.
- [ ] Implement a Hail Core detector: Search for areas of high reflectivity (>55 dBZ) co-located with low CC (< 0.95) and near-zero ZDR.
- [ ] **Storm Tracking (`/internal/analysis/tracking.go`):**
- [ ] Implement a centroid tracking algorithm.
- [ ] For each frame, calculate the center-of-mass for each `StormCell`.
- [ ] Correlate cells between frames by finding the closest centroid from the previous frame.
- [ ] Assign a persistent `TrackID` to each storm. Store the storm's history (location, time, intensity) in memory.
- [ ] **Threat Generation (`/internal/analysis/threats.go`):**
- [ ] Define a `Threat` struct: `TrackID`, `ThreatType` (Tornado, Hail), `Severity`, `PredictedPath` (a GeoJSON line or polygon).
- [ ] Create a rules engine that ingests `StormCell` objects and generates `Threat` objects.
- [ ] Example Rule: IF `StormCell.HasTVS == true` AND `StormCell.Intensity > Threshold`, THEN generate a "Tornado Warning" threat object.
- [ ] Extrapolate a simple predicted path based on the storm's recent movement vector.
- [ ] **Storm Cell Identification (`/internal/analysis/segmentation.go`)**
- [ ] Flood-fill or connected-components blob detection on composite reflectivity
- [ ] Output: list of `StormCell` objects with bounding polygon, centroid, max dBZ
- [ ] **Storm Tracking (`/internal/analysis/tracking.go`)**
- [ ] Frame-to-frame centroid correlation to assign persistent `TrackID`
- [ ] Store per-track history: location, time, intensity, motion vector
- [ ] **Velocity Source**
- [ ] Investigate alternative velocity product paths on NWS FTP (NWS 403 on `DS.p99r0`)
- [ ] Or ingest Level 2 data from `noaa-nexrad-level2` S3 (requester-pays, need creds)
- [ ] Parse velocity, ZDR, CC fields once a source is confirmed
- [ ] **Signature Detection (`/internal/analysis/signatures.go`)**
- [ ] TVS detector: strong inbound/outbound velocity couplet within a cell
- [ ] Hail core detector: dBZ > 55 co-located with low CC and near-zero ZDR
- [ ] **Threat Engine (`/internal/analysis/threats.go`)**
- [ ] `Threat` struct: TrackID, type, severity, predicted path GeoJSON
- [ ] Rules engine: TVS + intensity threshold → tornado threat object
- [ ] Extrapolate predicted path from recent motion vector
---
## Phase 4: Frontend Refinement with HTMX
## Phase 4: Frontend Refinement
*Goal: Build a dynamic, interactive, and useful user interface without a heavy JS framework.*
*Goal: Make the analysis actionable in the UI.*
- [ ] **Create API Endpoints for UI Components:**
- [ ] `/ui/storm-list`: Returns an HTML fragment (`<ul>...</ul>`) of currently active tracked storms.
- [ ] `/ui/storm-detail/{track_id}`: Returns an HTML fragment (`<div>...</div>`) with detailed stats for a specific storm.
- [ ] **Dynamic UI with HTMX:**
- [ ] Create a sidebar on the main page. Use `hx-get="/ui/storm-list"` and `hx-trigger="every 15s"` to keep the list of active storms up-to-date.
- [ ] Make each item in the storm list clickable. Use `hx-get="/ui/storm-detail/{track_id}"` and `hx-target="#detail-pane"` to load storm details into another panel without a page reload.
- [ ] **Map Interactivity:**
- [ ] When a storm detail is loaded, add a GeoJSON layer to the Leaflet map showing its current position and predicted track.
- [ ] Use a small vanilla JavaScript `htmx:afterSwap` event listener to command the Leaflet map to pan and zoom to the selected storm's coordinates (passed as `data-` attributes on the swapped HTML fragment).
- [ ] Create a search bar that allows a user to input an address. Use an external geocoding API to get coordinates and place a marker on the map.
- [ ] Storm list sidebar (HTMX, refreshes every 15 s)
- [ ] Storm detail panel on click (centroid, max dBZ, motion, TVS flag)
- [ ] Map overlay: GeoJSON storm polygons + predicted track line
- [ ] Pan/zoom to selected storm on sidebar click
- [ ] Address search bar with geocoding → drop pin on map
- [ ] Opacity slider for radar overlay
- [ ] Mobile-responsive layout
---
## Phase 5: SRE & Productionization
*Goal: Ensure the system is reliable, scalable, and observable, ready for real-world use.*
- [ ] **Containerization:**
- [ ] Write a multi-stage `Dockerfile`. The first stage builds the Go binary, the second copies the binary into a minimal `distroless` or `alpine` base image for a small, secure result.
- [ ] **Configuration Management:**
- [ ] Move all hardcoded settings (radar lists, thresholds) into a configuration file (e.g., `config.yaml`) or environment variables. Use a library like Viper to manage this.
- [ ] **Deployment (Docker Compose):**
- [ ] Write a `docker-compose.yml` that starts the `heloha-server` container alongside a Prometheus container scraping its `/metrics` endpoint.
- [ ] Implement proper liveness (`/healthz`) and readiness probes in your Go server. The readiness check should fail if no radar data has been successfully processed recently.
- [ ] *Future path: when ready to scale, migrate to Kubernetes manifests (Deployment, Service, Ingress) packaged with Helm. The Dockerfile and `/healthz`/readiness endpoints you build now translate directly.*
- [ ] **Observability:**
- [ ] **Metrics:** Add the `prometheus/client_golang` library. Instrument your code to export key metrics:
- `heloha_radar_files_processed_total{radar="KTLX", status="success"}`
- `heloha_data_latency_seconds` (time from file timestamp to processing completion)
- `heloha_active_threats_gauge`
- [ ] **Logging:** Use Go's stdlib `slog` with JSON output. Log key events (new threat detected, radar feed lost, tile cache stats).
- [ ] **Alerting:** Configure Prometheus and Alertmanager (included in the Compose stack). Create critical alerts:
- `ALERT RadarFeedDown IF (time() - last_success_timestamp{radar="KTLX"}) > 600`
- `ALERT HighProcessingLatency`
- `ALERT AppDown (based on container health check)`
- [ ] Multi-stage `Dockerfile` (build → distroless)
- [ ] `docker-compose.yml` with Prometheus scraping `/metrics`
- [ ] Prometheus metrics: `radar_files_processed_total`, `data_latency_seconds`, `active_threats_gauge`
- [ ] Readiness probe: fail if no radar data ingested in last 10 min
- [ ] Alertmanager rules: `RadarFeedDown`, `HighProcessingLatency`, `AppDown`
- [ ] Config file (`config.yaml` or env vars) for site list, thresholds, ports
---
## Backlog / Future Ideas
- [ ] **Mesonet Integration:** Add another data ingestion pipeline for the Oklahoma Mesonet. Correlate ground-level wind speed/pressure drops with radar signatures.
- [ ] **Database Persistence:** Store historical storm tracks and threat polygons in a time-series or geospatial database (e.g., TimescaleDB, PostGIS).
- [ ] **Push Notifications:** Implement a system (e.g., WebSockets or a mobile app) to send real-time alerts to users based on their registered location.
- [ ] **Machine Learning:** Train a model on historical radar data and confirmed tornado reports to create a more accurate probabilistic threat model.
- [ ] Oklahoma Mesonet integration (ground-level wind, pressure, temp)
- [ ] Database persistence for storm tracks (TimescaleDB or PostGIS)
- [ ] WebSocket push notifications for threats near a user's location
- [ ] ML model trained on historical radar + confirmed tornado reports
- [ ] Animate warning polygon borders (pulsing CSS) for active tornado warnings
- [ ] Multi-tilt display (0.5°, 1.5°, 2.5°) — currently fixed at 0.5°