blakeridgway/heloha
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
Blake Ridgway
2026-04-24 19:29:37 -05:00
commit 0c1547e0d5
3 changed files with 231 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
.gitignore vendored Normal file
View File

@@ -0,0 +1,23 @@
# Build output
bin/
dist/
# Go
*.exe
*.test
*.out
vendor/
# Config (may contain secrets)
config.yaml
config.local.yaml
.env
# OS
.DS_Store
Thumbs.db
# Editor
.vscode/
.idea/
*.swp

39
README.md Normal file
View File

@@ -0,0 +1,39 @@
# Heloha
> **Work in Progress** — this project is in early development. Nothing here is production-ready.
---
## What is this?
Heloha is a personal severe weather analysis and alerting platform built specifically for Oklahoma. It pulls raw radar data directly from NOAA's public archives, processes it, and presents a unified, real-time picture of what's happening in the sky — without relying on a third-party weather service to tell you what to think about it.
The name comes from the Choctaw word for thunder.
---
## Why build this?
Oklahoma sits in one of the most active severe weather corridors in the world. There are excellent commercial tools for watching weather, but they are built for a general audience. They smooth over the details, apply conservative thresholds, and optimize for not alarming people unnecessarily.
That's the right call for most users. It's not the right call when you want to understand what a storm is actually doing.
The goal here is to work directly with the underlying data — the same WSR-88D NEXRAD feeds that professional meteorologists use — and build tooling that answers the question a commercial app won't: *is this storm becoming dangerous right now, and where is it going?*
---
## What we're trying to accomplish
- **Ingest and parse** real-time NEXRAD radar data for all Oklahoma-area radar sites (KTLX, KINX, KFDR, KVNX, and others)
- **Fuse multiple radars** into a single, seamless statewide picture, always using the best available coverage for any given point on the ground
- **Identify and track storm cells** automatically, following them across radar scans and building a history of their movement and intensity
- **Detect severe signatures** — tornadic vortex signatures (TVS) from velocity data, hail cores from differential reflectivity — and surface those as structured, actionable threat objects
- **Present this cleanly** in a browser-based map interface that updates in real time, with no heavy JavaScript framework required
The end state is a platform that gives a single operator a clear, honest read on severe weather across the state — updated every few minutes, with enough analytical depth to be genuinely useful during an active weather event.
---
## Stack
Go · HTMX · Leaflet.js · NOAA NEXRAD (AWS S3) · Docker Compose · Prometheus

169
TODO.md Normal file
View File

@@ -0,0 +1,169 @@
# Project Heloha
A detailed, actionable task list for building a unified, multi-radar severe weather analysis and alerting platform for Oklahoma.
---
## Phase 0: Foundation & Setup (The First Week)
*Goal: Establish a clean, professional Go project environment.*
- [x] **Initialize Git Repository:**
- [x] `git init`
- [x] Create a `.gitignore` file for Go and common OS files.
- [x] Create a `README.md` with the project's mission statement.
- [ ] **Establish Go Project Structure:**
- [ ] `go mod init github.com/blakeridgway/heloha`
- [ ] Create a standard Go project layout:
- `/cmd/heloha-server`: Main application entry point.
- `/internal/radar`: Code for fetching, parsing, and processing radar data.
- `/internal/server`: HTTP handlers and server logic.
- `/internal/config`: Configuration management.
- `/web/templates`: HTML templates for the frontend.
- `/web/static`: CSS and JavaScript assets.
- [ ] **Create a Basic Web Server:**
- [ ] In `/cmd/heloha-server/main.go`, set up an `http.Server`.
- [ ] Choose and implement a router (e.g., `go-chi/chi` is a good choice for middleware and flexibility).
- [ ] Create a simple `/healthz` endpoint that returns `200 OK`.
- [ ] **Set Up Build Automation:**
- [ ] Create a `Makefile` with targets for:
- `build`: `go build -o bin/heloha-server ./cmd/heloha-server`
- `run`: `go run ./cmd/heloha-server`
- `test`: `go test ./...`
- `tidy`: `go mod tidy`
---
## Phase 1: Single Radar Ingestion & Display (The MVP)
*Goal: Get raw data from one radar onto a map on a webpage. This proves the core data pipeline is viable.*
- [ ] **Data Acquisition (`/internal/radar/fetch.go`):**
- [ ] Add the AWS SDK for Go V2 (`aws-sdk-go-v2`) as a dependency.
- [ ] **Use NEXRAD Level 3 data** from the `unidata-nexrad-level3` S3 bucket. Level 3 products are pre-processed by the RPG into a simple binary format — no polar-to-Cartesian projection needed, and no per-record bzip2 complexity.
- [ ] Implement logic to find the *latest* file for a given radar (`KTLX`) and product code (`N0B` = Base Reflectivity 0.5° tilt) using the S3 key naming convention.
- [ ] Write a function to download a specific Level 3 file from S3 into memory.
- [ ] *Note: Level 2 data (raw I/Q + full velocity fields) will be introduced in Phase 3 when TVS detection requires it. Level 3 is sufficient for reflectivity display and mosaic.*
- [ ] **Data Parsing (`/internal/radar/parse.go`):**
- [ ] Implement a Level 3 product parser targeting the Graphic Product Message format (ICD 2620001). The header is fixed-width; the symbology block contains pre-gridded radial data.
- [ ] Define Go `structs` to hold the parsed data: a `RadarProduct` struct with base reflectivity (a 2D array of floats), timestamp, elevation angle, and radar site coordinates.
- [ ] **Visualization Backend (`/internal/server/handlers.go`):**
- [ ] Create a new HTTP handler for map tiles: `/api/v1/tile/ktlx/{z}/{x}/{y}.png`.
- [ ] Inside the handler, implement logic to:
- Load the latest parsed `KTLX` data.
- Use the Z/X/Y tile coordinates to determine the required geographic bounds.
- Map the pre-gridded radar data onto the requested tile canvas (no polar projection needed at this stage).
- Use a color lookup table (LUT) to convert dBZ values to colors (e.g., green for light rain, red for heavy).
- Render the final tile as a PNG image using Go's standard `image` package.
- [ ] **Tile Caching (`/internal/server/tilecache.go`):**
- [ ] Implement an in-memory tile cache as a `map[string][]byte` (key: `"z/x/y"`) protected by a `sync.RWMutex`.
- [ ] Wrap the tile handler: check the cache first; on a miss, render the PNG, store it in the cache, and return it.
- [ ] Tie cache invalidation to the data update cycle — when a new `RadarProduct` is ingested, clear the cache for that radar. This ensures tiles are never stale beyond one update interval.
- [ ] **Frontend Display (`/web/templates/index.html`):**
- [ ] Set up a basic HTML page that includes Leaflet.js from a CDN.
- [ ] Initialize a Leaflet map centered on Oklahoma.
- [ ] Add a `L.tileLayer` pointing to your new `/api/v1/tile/ktlx/{z}/{x}/{y}.png` endpoint.
- [ ] Add the HTMX script tag.
- [ ] Use `hx-get` and `hx-trigger="every 60s"` on a container element to periodically refresh the map layer or associated metadata.
---
## Phase 2: Multi-Radar Fusion (The Core Challenge)
*Goal: Transition from a single, siloed view to a single, authoritative statewide weather picture.*
- [ ] **Concurrent Ingestion (`/internal/radar/manager.go`):**
- [ ] Design a concurrent manager that spawns one "fetcher" goroutine per radar (KTLX, KINX, KFDR, KVNX, etc.).
- [ ] Each fetcher goroutine periodically checks for new data for its assigned radar.
- [ ] Use a buffered Go `channel` to pass newly downloaded and parsed `RadarProduct` objects to a central processing component.
- [ ] **Data Mosaicing (`/internal/radar/mosaic.go`):**
- [ ] **Grid Definition:** Define a constant statewide grid in your code. This includes its geographic boundaries (min/max lat/lon), resolution (e.g., 1km x 1km), and dimensions (width/height in pixels).
- [ ] **Projection Logic:** Implement robust math functions to convert from (radar, azimuth, range) -> (lat, lon) -> (grid X, grid Y). This is crucial.
- [ ] **Compositing Engine:**
- Create a function that runs in a loop, triggered by new data or a timer.
- This function initializes a new, empty statewide grid.
- It iterates through every cell of the grid. For each cell, it checks all available radars to see which one provides the best data (lowest beam altitude above that grid point).
- It then "paints" the data from the best radar onto that grid cell.
- [ ] **State Management:** Store the latest completed mosaic grid in memory, protected by a mutex, so it can be safely accessed by the tile-serving handler.
- [ ] **Update Visualization Backend:**
- [ ] Modify the `/api/v1/tile/...` handler to render tiles from the statewide mosaic grid instead of a single radar product.
- [ ] The tile handler no longer needs the radar ID in the URL. New URL: `/api/v1/tile/reflectivity/{z}/{x}/{y}.png`.
---
## Phase 3: Advanced Analysis & Alerting
*Goal: Find the "so what?" in the data. Identify and track dangerous storms.*
- [ ] **Expand Data Parsing (introduce Level 2):**
- [ ] Add Level 2 ingestion from the `noaa-nexrad-level2` S3 bucket alongside the existing Level 3 pipeline. Level 2 files are compressed with bzip2 on a per-record basis — implement record-level decompression.
- [ ] Parse Level 2 Velocity, Differential Reflectivity (ZDR), and Correlation Coefficient (CC) data fields. The official format spec is ICD RDA/RPG 2620002U.
- [ ] Update the mosaicing engine to create composite grids for these new data types.
- [ ] **Storm Cell Identification (`/internal/analysis/segmentation.go`):**
- [ ] Implement a "blob detection" algorithm on the composite reflectivity grid. This can be a simple algorithm like flood-fill or a more advanced one like DBSCAN.
- [ ] The output should be a list of distinct `StormCell` objects, each with a polygon defining its boundary.
- [ ] **Signature Detection (`/internal/analysis/signatures.go`):**
- [ ] For each identified `StormCell`, analyze the underlying data within its polygon.
- [ ] Implement a TVS (Tornadic Vortex Signature) detector: Search for strong, compact, inbound/outbound velocity couplets.
- [ ] Implement a Hail Core detector: Search for areas of high reflectivity (>55 dBZ) co-located with low CC (< 0.95) and near-zero ZDR.
- [ ] **Storm Tracking (`/internal/analysis/tracking.go`):**
- [ ] Implement a centroid tracking algorithm.
- [ ] For each frame, calculate the center-of-mass for each `StormCell`.
- [ ] Correlate cells between frames by finding the closest centroid from the previous frame.
- [ ] Assign a persistent `TrackID` to each storm. Store the storm's history (location, time, intensity) in memory.
- [ ] **Threat Generation (`/internal/analysis/threats.go`):**
- [ ] Define a `Threat` struct: `TrackID`, `ThreatType` (Tornado, Hail), `Severity`, `PredictedPath` (a GeoJSON line or polygon).
- [ ] Create a rules engine that ingests `StormCell` objects and generates `Threat` objects.
- [ ] Example Rule: IF `StormCell.HasTVS == true` AND `StormCell.Intensity > Threshold`, THEN generate a "Tornado Warning" threat object.
- [ ] Extrapolate a simple predicted path based on the storm's recent movement vector.
---
## Phase 4: Frontend Refinement with HTMX
*Goal: Build a dynamic, interactive, and useful user interface without a heavy JS framework.*
- [ ] **Create API Endpoints for UI Components:**
- [ ] `/ui/storm-list`: Returns an HTML fragment (`<ul>...</ul>`) of currently active tracked storms.
- [ ] `/ui/storm-detail/{track_id}`: Returns an HTML fragment (`<div>...</div>`) with detailed stats for a specific storm.
- [ ] **Dynamic UI with HTMX:**
- [ ] Create a sidebar on the main page. Use `hx-get="/ui/storm-list"` and `hx-trigger="every 15s"` to keep the list of active storms up-to-date.
- [ ] Make each item in the storm list clickable. Use `hx-get="/ui/storm-detail/{track_id}"` and `hx-target="#detail-pane"` to load storm details into another panel without a page reload.
- [ ] **Map Interactivity:**
- [ ] When a storm detail is loaded, add a GeoJSON layer to the Leaflet map showing its current position and predicted track.
- [ ] Use a small vanilla JavaScript `htmx:afterSwap` event listener to command the Leaflet map to pan and zoom to the selected storm's coordinates (passed as `data-` attributes on the swapped HTML fragment).
- [ ] Create a search bar that allows a user to input an address. Use an external geocoding API to get coordinates and place a marker on the map.
---
## Phase 5: SRE & Productionization
*Goal: Ensure the system is reliable, scalable, and observable, ready for real-world use.*
- [ ] **Containerization:**
- [ ] Write a multi-stage `Dockerfile`. The first stage builds the Go binary, the second copies the binary into a minimal `distroless` or `alpine` base image for a small, secure result.
- [ ] **Configuration Management:**
- [ ] Move all hardcoded settings (radar lists, thresholds) into a configuration file (e.g., `config.yaml`) or environment variables. Use a library like Viper to manage this.
- [ ] **Deployment (Docker Compose):**
- [ ] Write a `docker-compose.yml` that starts the `heloha-server` container alongside a Prometheus container scraping its `/metrics` endpoint.
- [ ] Implement proper liveness (`/healthz`) and readiness probes in your Go server. The readiness check should fail if no radar data has been successfully processed recently.
- [ ] *Future path: when ready to scale, migrate to Kubernetes manifests (Deployment, Service, Ingress) packaged with Helm. The Dockerfile and `/healthz`/readiness endpoints you build now translate directly.*
- [ ] **Observability:**
- [ ] **Metrics:** Add the `prometheus/client_golang` library. Instrument your code to export key metrics:
- `heloha_radar_files_processed_total{radar="KTLX", status="success"}`
- `heloha_data_latency_seconds` (time from file timestamp to processing completion)
- `heloha_active_threats_gauge`
- [ ] **Logging:** Use Go's stdlib `slog` with JSON output. Log key events (new threat detected, radar feed lost, tile cache stats).
- [ ] **Alerting:** Configure Prometheus and Alertmanager (included in the Compose stack). Create critical alerts:
- `ALERT RadarFeedDown IF (time() - last_success_timestamp{radar="KTLX"}) > 600`
- `ALERT HighProcessingLatency`
- `ALERT AppDown (based on container health check)`
---
## Backlog / Future Ideas
- [ ] **Mesonet Integration:** Add another data ingestion pipeline for the Oklahoma Mesonet. Correlate ground-level wind speed/pressure drops with radar signatures.
- [ ] **Database Persistence:** Store historical storm tracks and threat polygons in a time-series or geospatial database (e.g., TimescaleDB, PostGIS).
- [ ] **Push Notifications:** Implement a system (e.g., WebSockets or a mobile app) to send real-time alerts to users based on their registered location.
- [ ] **Machine Learning:** Train a model on historical radar data and confirmed tornado reports to create a more accurate probabilistic threat model.