From 0c1547e0d568dfc2a35b99a469729b60b3044bd9 Mon Sep 17 00:00:00 2001 From: Blake Ridgway Date: Fri, 24 Apr 2026 19:29:37 -0500 Subject: [PATCH] first commit --- .gitignore | 23 ++++++++ README.md | 39 +++++++++++++ TODO.md | 169 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 231 insertions(+) create mode 100644 .gitignore create mode 100644 README.md create mode 100644 TODO.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..fc63ccb --- /dev/null +++ b/.gitignore @@ -0,0 +1,23 @@ +# Build output +bin/ +dist/ + +# Go +*.exe +*.test +*.out +vendor/ + +# Config (may contain secrets) +config.yaml +config.local.yaml +.env + +# OS +.DS_Store +Thumbs.db + +# Editor +.vscode/ +.idea/ +*.swp diff --git a/README.md b/README.md new file mode 100644 index 0000000..a874dac --- /dev/null +++ b/README.md @@ -0,0 +1,39 @@ +# Heloha + +> **Work in Progress** — this project is in early development. Nothing here is production-ready. + +--- + +## What is this? + +Heloha is a personal severe weather analysis and alerting platform built specifically for Oklahoma. It pulls raw radar data directly from NOAA's public archives, processes it, and presents a unified, real-time picture of what's happening in the sky — without relying on a third-party weather service to tell you what to think about it. + +The name comes from the Choctaw word for thunder. + +--- + +## Why build this? + +Oklahoma sits in one of the most active severe weather corridors in the world. There are excellent commercial tools for watching weather, but they are built for a general audience. They smooth over the details, apply conservative thresholds, and optimize for not alarming people unnecessarily. + +That's the right call for most users. It's not the right call when you want to understand what a storm is actually doing. + +The goal here is to work directly with the underlying data — the same WSR-88D NEXRAD feeds that professional meteorologists use — and build tooling that answers the question a commercial app won't: *is this storm becoming dangerous right now, and where is it going?* + +--- + +## What we're trying to accomplish + +- **Ingest and parse** real-time NEXRAD radar data for all Oklahoma-area radar sites (KTLX, KINX, KFDR, KVNX, and others) +- **Fuse multiple radars** into a single, seamless statewide picture, always using the best available coverage for any given point on the ground +- **Identify and track storm cells** automatically, following them across radar scans and building a history of their movement and intensity +- **Detect severe signatures** — tornadic vortex signatures (TVS) from velocity data, hail cores from differential reflectivity — and surface those as structured, actionable threat objects +- **Present this cleanly** in a browser-based map interface that updates in real time, with no heavy JavaScript framework required + +The end state is a platform that gives a single operator a clear, honest read on severe weather across the state — updated every few minutes, with enough analytical depth to be genuinely useful during an active weather event. + +--- + +## Stack + +Go · HTMX · Leaflet.js · NOAA NEXRAD (AWS S3) · Docker Compose · Prometheus diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..24e91ba --- /dev/null +++ b/TODO.md @@ -0,0 +1,169 @@ +# Project Heloha + +A detailed, actionable task list for building a unified, multi-radar severe weather analysis and alerting platform for Oklahoma. + +--- + +## Phase 0: Foundation & Setup (The First Week) + +*Goal: Establish a clean, professional Go project environment.* + +- [x] **Initialize Git Repository:** + - [x] `git init` + - [x] Create a `.gitignore` file for Go and common OS files. + - [x] Create a `README.md` with the project's mission statement. +- [ ] **Establish Go Project Structure:** + - [ ] `go mod init github.com/blakeridgway/heloha` + - [ ] Create a standard Go project layout: + - `/cmd/heloha-server`: Main application entry point. + - `/internal/radar`: Code for fetching, parsing, and processing radar data. + - `/internal/server`: HTTP handlers and server logic. + - `/internal/config`: Configuration management. + - `/web/templates`: HTML templates for the frontend. + - `/web/static`: CSS and JavaScript assets. +- [ ] **Create a Basic Web Server:** + - [ ] In `/cmd/heloha-server/main.go`, set up an `http.Server`. + - [ ] Choose and implement a router (e.g., `go-chi/chi` is a good choice for middleware and flexibility). + - [ ] Create a simple `/healthz` endpoint that returns `200 OK`. +- [ ] **Set Up Build Automation:** + - [ ] Create a `Makefile` with targets for: + - `build`: `go build -o bin/heloha-server ./cmd/heloha-server` + - `run`: `go run ./cmd/heloha-server` + - `test`: `go test ./...` + - `tidy`: `go mod tidy` + +--- + +## Phase 1: Single Radar Ingestion & Display (The MVP) + +*Goal: Get raw data from one radar onto a map on a webpage. This proves the core data pipeline is viable.* + +- [ ] **Data Acquisition (`/internal/radar/fetch.go`):** + - [ ] Add the AWS SDK for Go V2 (`aws-sdk-go-v2`) as a dependency. + - [ ] **Use NEXRAD Level 3 data** from the `unidata-nexrad-level3` S3 bucket. Level 3 products are pre-processed by the RPG into a simple binary format — no polar-to-Cartesian projection needed, and no per-record bzip2 complexity. + - [ ] Implement logic to find the *latest* file for a given radar (`KTLX`) and product code (`N0B` = Base Reflectivity 0.5° tilt) using the S3 key naming convention. + - [ ] Write a function to download a specific Level 3 file from S3 into memory. + - [ ] *Note: Level 2 data (raw I/Q + full velocity fields) will be introduced in Phase 3 when TVS detection requires it. Level 3 is sufficient for reflectivity display and mosaic.* +- [ ] **Data Parsing (`/internal/radar/parse.go`):** + - [ ] Implement a Level 3 product parser targeting the Graphic Product Message format (ICD 2620001). The header is fixed-width; the symbology block contains pre-gridded radial data. + - [ ] Define Go `structs` to hold the parsed data: a `RadarProduct` struct with base reflectivity (a 2D array of floats), timestamp, elevation angle, and radar site coordinates. +- [ ] **Visualization Backend (`/internal/server/handlers.go`):** + - [ ] Create a new HTTP handler for map tiles: `/api/v1/tile/ktlx/{z}/{x}/{y}.png`. + - [ ] Inside the handler, implement logic to: + - Load the latest parsed `KTLX` data. + - Use the Z/X/Y tile coordinates to determine the required geographic bounds. + - Map the pre-gridded radar data onto the requested tile canvas (no polar projection needed at this stage). + - Use a color lookup table (LUT) to convert dBZ values to colors (e.g., green for light rain, red for heavy). + - Render the final tile as a PNG image using Go's standard `image` package. +- [ ] **Tile Caching (`/internal/server/tilecache.go`):** + - [ ] Implement an in-memory tile cache as a `map[string][]byte` (key: `"z/x/y"`) protected by a `sync.RWMutex`. + - [ ] Wrap the tile handler: check the cache first; on a miss, render the PNG, store it in the cache, and return it. + - [ ] Tie cache invalidation to the data update cycle — when a new `RadarProduct` is ingested, clear the cache for that radar. This ensures tiles are never stale beyond one update interval. +- [ ] **Frontend Display (`/web/templates/index.html`):** + - [ ] Set up a basic HTML page that includes Leaflet.js from a CDN. + - [ ] Initialize a Leaflet map centered on Oklahoma. + - [ ] Add a `L.tileLayer` pointing to your new `/api/v1/tile/ktlx/{z}/{x}/{y}.png` endpoint. + - [ ] Add the HTMX script tag. + - [ ] Use `hx-get` and `hx-trigger="every 60s"` on a container element to periodically refresh the map layer or associated metadata. + +--- + +## Phase 2: Multi-Radar Fusion (The Core Challenge) + +*Goal: Transition from a single, siloed view to a single, authoritative statewide weather picture.* + +- [ ] **Concurrent Ingestion (`/internal/radar/manager.go`):** + - [ ] Design a concurrent manager that spawns one "fetcher" goroutine per radar (KTLX, KINX, KFDR, KVNX, etc.). + - [ ] Each fetcher goroutine periodically checks for new data for its assigned radar. + - [ ] Use a buffered Go `channel` to pass newly downloaded and parsed `RadarProduct` objects to a central processing component. +- [ ] **Data Mosaicing (`/internal/radar/mosaic.go`):** + - [ ] **Grid Definition:** Define a constant statewide grid in your code. This includes its geographic boundaries (min/max lat/lon), resolution (e.g., 1km x 1km), and dimensions (width/height in pixels). + - [ ] **Projection Logic:** Implement robust math functions to convert from (radar, azimuth, range) -> (lat, lon) -> (grid X, grid Y). This is crucial. + - [ ] **Compositing Engine:** + - Create a function that runs in a loop, triggered by new data or a timer. + - This function initializes a new, empty statewide grid. + - It iterates through every cell of the grid. For each cell, it checks all available radars to see which one provides the best data (lowest beam altitude above that grid point). + - It then "paints" the data from the best radar onto that grid cell. + - [ ] **State Management:** Store the latest completed mosaic grid in memory, protected by a mutex, so it can be safely accessed by the tile-serving handler. +- [ ] **Update Visualization Backend:** + - [ ] Modify the `/api/v1/tile/...` handler to render tiles from the statewide mosaic grid instead of a single radar product. + - [ ] The tile handler no longer needs the radar ID in the URL. New URL: `/api/v1/tile/reflectivity/{z}/{x}/{y}.png`. + +--- + +## Phase 3: Advanced Analysis & Alerting + +*Goal: Find the "so what?" in the data. Identify and track dangerous storms.* + +- [ ] **Expand Data Parsing (introduce Level 2):** + - [ ] Add Level 2 ingestion from the `noaa-nexrad-level2` S3 bucket alongside the existing Level 3 pipeline. Level 2 files are compressed with bzip2 on a per-record basis — implement record-level decompression. + - [ ] Parse Level 2 Velocity, Differential Reflectivity (ZDR), and Correlation Coefficient (CC) data fields. The official format spec is ICD RDA/RPG 2620002U. + - [ ] Update the mosaicing engine to create composite grids for these new data types. +- [ ] **Storm Cell Identification (`/internal/analysis/segmentation.go`):** + - [ ] Implement a "blob detection" algorithm on the composite reflectivity grid. This can be a simple algorithm like flood-fill or a more advanced one like DBSCAN. + - [ ] The output should be a list of distinct `StormCell` objects, each with a polygon defining its boundary. +- [ ] **Signature Detection (`/internal/analysis/signatures.go`):** + - [ ] For each identified `StormCell`, analyze the underlying data within its polygon. + - [ ] Implement a TVS (Tornadic Vortex Signature) detector: Search for strong, compact, inbound/outbound velocity couplets. + - [ ] Implement a Hail Core detector: Search for areas of high reflectivity (>55 dBZ) co-located with low CC (< 0.95) and near-zero ZDR. +- [ ] **Storm Tracking (`/internal/analysis/tracking.go`):** + - [ ] Implement a centroid tracking algorithm. + - [ ] For each frame, calculate the center-of-mass for each `StormCell`. + - [ ] Correlate cells between frames by finding the closest centroid from the previous frame. + - [ ] Assign a persistent `TrackID` to each storm. Store the storm's history (location, time, intensity) in memory. +- [ ] **Threat Generation (`/internal/analysis/threats.go`):** + - [ ] Define a `Threat` struct: `TrackID`, `ThreatType` (Tornado, Hail), `Severity`, `PredictedPath` (a GeoJSON line or polygon). + - [ ] Create a rules engine that ingests `StormCell` objects and generates `Threat` objects. + - [ ] Example Rule: IF `StormCell.HasTVS == true` AND `StormCell.Intensity > Threshold`, THEN generate a "Tornado Warning" threat object. + - [ ] Extrapolate a simple predicted path based on the storm's recent movement vector. + +--- + +## Phase 4: Frontend Refinement with HTMX + +*Goal: Build a dynamic, interactive, and useful user interface without a heavy JS framework.* + +- [ ] **Create API Endpoints for UI Components:** + - [ ] `/ui/storm-list`: Returns an HTML fragment (``) of currently active tracked storms. + - [ ] `/ui/storm-detail/{track_id}`: Returns an HTML fragment (`
...
`) with detailed stats for a specific storm. +- [ ] **Dynamic UI with HTMX:** + - [ ] Create a sidebar on the main page. Use `hx-get="/ui/storm-list"` and `hx-trigger="every 15s"` to keep the list of active storms up-to-date. + - [ ] Make each item in the storm list clickable. Use `hx-get="/ui/storm-detail/{track_id}"` and `hx-target="#detail-pane"` to load storm details into another panel without a page reload. +- [ ] **Map Interactivity:** + - [ ] When a storm detail is loaded, add a GeoJSON layer to the Leaflet map showing its current position and predicted track. + - [ ] Use a small vanilla JavaScript `htmx:afterSwap` event listener to command the Leaflet map to pan and zoom to the selected storm's coordinates (passed as `data-` attributes on the swapped HTML fragment). + - [ ] Create a search bar that allows a user to input an address. Use an external geocoding API to get coordinates and place a marker on the map. + +--- + +## Phase 5: SRE & Productionization + +*Goal: Ensure the system is reliable, scalable, and observable, ready for real-world use.* + +- [ ] **Containerization:** + - [ ] Write a multi-stage `Dockerfile`. The first stage builds the Go binary, the second copies the binary into a minimal `distroless` or `alpine` base image for a small, secure result. +- [ ] **Configuration Management:** + - [ ] Move all hardcoded settings (radar lists, thresholds) into a configuration file (e.g., `config.yaml`) or environment variables. Use a library like Viper to manage this. +- [ ] **Deployment (Docker Compose):** + - [ ] Write a `docker-compose.yml` that starts the `heloha-server` container alongside a Prometheus container scraping its `/metrics` endpoint. + - [ ] Implement proper liveness (`/healthz`) and readiness probes in your Go server. The readiness check should fail if no radar data has been successfully processed recently. + - [ ] *Future path: when ready to scale, migrate to Kubernetes manifests (Deployment, Service, Ingress) packaged with Helm. The Dockerfile and `/healthz`/readiness endpoints you build now translate directly.* +- [ ] **Observability:** + - [ ] **Metrics:** Add the `prometheus/client_golang` library. Instrument your code to export key metrics: + - `heloha_radar_files_processed_total{radar="KTLX", status="success"}` + - `heloha_data_latency_seconds` (time from file timestamp to processing completion) + - `heloha_active_threats_gauge` + - [ ] **Logging:** Use Go's stdlib `slog` with JSON output. Log key events (new threat detected, radar feed lost, tile cache stats). + - [ ] **Alerting:** Configure Prometheus and Alertmanager (included in the Compose stack). Create critical alerts: + - `ALERT RadarFeedDown IF (time() - last_success_timestamp{radar="KTLX"}) > 600` + - `ALERT HighProcessingLatency` + - `ALERT AppDown (based on container health check)` + +--- + +## Backlog / Future Ideas + +- [ ] **Mesonet Integration:** Add another data ingestion pipeline for the Oklahoma Mesonet. Correlate ground-level wind speed/pressure drops with radar signatures. +- [ ] **Database Persistence:** Store historical storm tracks and threat polygons in a time-series or geospatial database (e.g., TimescaleDB, PostGIS). +- [ ] **Push Notifications:** Implement a system (e.g., WebSockets or a mobile app) to send real-time alerts to users based on their registered location. +- [ ] **Machine Learning:** Train a model on historical radar data and confirmed tornado reports to create a more accurate probabilistic threat model.