Project Heloha

A detailed, actionable task list for building a unified, multi-radar severe weather analysis and alerting platform for Oklahoma.

Phase 0: Foundation & Setup (The First Week)

Goal: Establish a clean, professional Go project environment.

Initialize Git Repository:
- git init
- Create a .gitignore file for Go and common OS files.
- Create a README.md with the project's mission statement.
Establish Go Project Structure:
- go mod init github.com/blakeridgway/heloha
- Create a standard Go project layout:
  - /cmd/heloha-server: Main application entry point.
  - /internal/radar: Code for fetching, parsing, and processing radar data.
  - /internal/server: HTTP handlers and server logic.
  - /internal/config: Configuration management.
  - /web/templates: HTML templates for the frontend.
  - /web/static: CSS and JavaScript assets.
Create a Basic Web Server:
- In /cmd/heloha-server/main.go, set up an http.Server.
- Choose and implement a router (e.g., go-chi/chi is a good choice for middleware and flexibility).
- Create a simple /healthz endpoint that returns 200 OK.
Set Up Build Automation:
- Create a Makefile with targets for:
  - build: go build -o bin/heloha-server ./cmd/heloha-server
  - run: go run ./cmd/heloha-server
  - test: go test ./...
  - tidy: go mod tidy

Phase 1: Single Radar Ingestion & Display (The MVP)

Goal: Get raw data from one radar onto a map on a webpage. This proves the core data pipeline is viable.

Data Acquisition (/internal/radar/fetch.go):
- Add the AWS SDK for Go V2 (aws-sdk-go-v2) as a dependency.
- Use NEXRAD Level 3 data from the unidata-nexrad-level3 S3 bucket. Level 3 products are pre-processed by the RPG into a simple binary format — no polar-to-Cartesian projection needed, and no per-record bzip2 complexity.
- Implement logic to find the latest file for a given radar (KTLX) and product code (N0B = Base Reflectivity 0.5° tilt) using the S3 key naming convention.
- Write a function to download a specific Level 3 file from S3 into memory.
- Note: Level 2 data (raw I/Q + full velocity fields) will be introduced in Phase 3 when TVS detection requires it. Level 3 is sufficient for reflectivity display and mosaic.
Data Parsing (/internal/radar/parse.go):
- Implement a Level 3 product parser targeting the Graphic Product Message format (ICD 2620001). The header is fixed-width; the symbology block contains pre-gridded radial data.
- Define Go structs to hold the parsed data: a RadarProduct struct with base reflectivity (a 2D array of floats), timestamp, elevation angle, and radar site coordinates.
Visualization Backend (/internal/server/handlers.go):
- Create a new HTTP handler for map tiles: /api/v1/tile/ktlx/{z}/{x}/{y}.png.
- Inside the handler, implement logic to:
  - Load the latest parsed KTLX data.
  - Use the Z/X/Y tile coordinates to determine the required geographic bounds.
  - Map the pre-gridded radar data onto the requested tile canvas (no polar projection needed at this stage).
  - Use a color lookup table (LUT) to convert dBZ values to colors (e.g., green for light rain, red for heavy).
  - Render the final tile as a PNG image using Go's standard image package.
Tile Caching (/internal/server/tilecache.go):
- Implement an in-memory tile cache as a map[string][]byte (key: "z/x/y") protected by a sync.RWMutex.
- Wrap the tile handler: check the cache first; on a miss, render the PNG, store it in the cache, and return it.
- Tie cache invalidation to the data update cycle — when a new RadarProduct is ingested, clear the cache for that radar. This ensures tiles are never stale beyond one update interval.
Frontend Display (/web/templates/index.html):
- Set up a basic HTML page that includes Leaflet.js from a CDN.
- Initialize a Leaflet map centered on Oklahoma.
- Add a L.tileLayer pointing to your new /api/v1/tile/ktlx/{z}/{x}/{y}.png endpoint.
- Add the HTMX script tag.
- Use hx-get and hx-trigger="every 60s" on a container element to periodically refresh the map layer or associated metadata.

Phase 2: Multi-Radar Fusion (The Core Challenge)

Goal: Transition from a single, siloed view to a single, authoritative statewide weather picture.

Concurrent Ingestion (/internal/radar/manager.go):
- Design a concurrent manager that spawns one "fetcher" goroutine per radar (KTLX, KINX, KFDR, KVNX, etc.).
- Each fetcher goroutine periodically checks for new data for its assigned radar.
- Use a buffered Go channel to pass newly downloaded and parsed RadarProduct objects to a central processing component.
Data Mosaicing (/internal/radar/mosaic.go):
- Grid Definition: Define a constant statewide grid in your code. This includes its geographic boundaries (min/max lat/lon), resolution (e.g., 1km x 1km), and dimensions (width/height in pixels).
- Projection Logic: Implement robust math functions to convert from (radar, azimuth, range) -> (lat, lon) -> (grid X, grid Y). This is crucial.
- Compositing Engine:
  - Create a function that runs in a loop, triggered by new data or a timer.
  - This function initializes a new, empty statewide grid.
  - It iterates through every cell of the grid. For each cell, it checks all available radars to see which one provides the best data (lowest beam altitude above that grid point).
  - It then "paints" the data from the best radar onto that grid cell.
- State Management: Store the latest completed mosaic grid in memory, protected by a mutex, so it can be safely accessed by the tile-serving handler.
Update Visualization Backend:
- Modify the /api/v1/tile/... handler to render tiles from the statewide mosaic grid instead of a single radar product.
- The tile handler no longer needs the radar ID in the URL. New URL: /api/v1/tile/reflectivity/{z}/{x}/{y}.png.

Phase 3: Advanced Analysis & Alerting

Goal: Find the "so what?" in the data. Identify and track dangerous storms.

Expand Data Parsing (introduce Level 2):
- Add Level 2 ingestion from the noaa-nexrad-level2 S3 bucket alongside the existing Level 3 pipeline. Level 2 files are compressed with bzip2 on a per-record basis — implement record-level decompression.
- Parse Level 2 Velocity, Differential Reflectivity (ZDR), and Correlation Coefficient (CC) data fields. The official format spec is ICD RDA/RPG 2620002U.
- Update the mosaicing engine to create composite grids for these new data types.
Storm Cell Identification (/internal/analysis/segmentation.go):
- Implement a "blob detection" algorithm on the composite reflectivity grid. This can be a simple algorithm like flood-fill or a more advanced one like DBSCAN.
- The output should be a list of distinct StormCell objects, each with a polygon defining its boundary.
Signature Detection (/internal/analysis/signatures.go):
- For each identified StormCell, analyze the underlying data within its polygon.
- Implement a TVS (Tornadic Vortex Signature) detector: Search for strong, compact, inbound/outbound velocity couplets.
- Implement a Hail Core detector: Search for areas of high reflectivity (>55 dBZ) co-located with low CC (< 0.95) and near-zero ZDR.
Storm Tracking (/internal/analysis/tracking.go):
- Implement a centroid tracking algorithm.
- For each frame, calculate the center-of-mass for each StormCell.
- Correlate cells between frames by finding the closest centroid from the previous frame.
- Assign a persistent TrackID to each storm. Store the storm's history (location, time, intensity) in memory.
Threat Generation (/internal/analysis/threats.go):
- Define a Threat struct: TrackID, ThreatType (Tornado, Hail), Severity, PredictedPath (a GeoJSON line or polygon).
- Create a rules engine that ingests StormCell objects and generates Threat objects.
- Example Rule: IF StormCell.HasTVS == true AND StormCell.Intensity > Threshold, THEN generate a "Tornado Warning" threat object.
- Extrapolate a simple predicted path based on the storm's recent movement vector.

Phase 4: Frontend Refinement with HTMX

Goal: Build a dynamic, interactive, and useful user interface without a heavy JS framework.

Create API Endpoints for UI Components:
- /ui/storm-list: Returns an HTML fragment (<ul>...</ul>) of currently active tracked storms.
- /ui/storm-detail/{track_id}: Returns an HTML fragment (<div>...</div>) with detailed stats for a specific storm.
Dynamic UI with HTMX:
- Create a sidebar on the main page. Use hx-get="/ui/storm-list" and hx-trigger="every 15s" to keep the list of active storms up-to-date.
- Make each item in the storm list clickable. Use hx-get="/ui/storm-detail/{track_id}" and hx-target="#detail-pane" to load storm details into another panel without a page reload.
Map Interactivity:
- When a storm detail is loaded, add a GeoJSON layer to the Leaflet map showing its current position and predicted track.
- Use a small vanilla JavaScript htmx:afterSwap event listener to command the Leaflet map to pan and zoom to the selected storm's coordinates (passed as data- attributes on the swapped HTML fragment).
- Create a search bar that allows a user to input an address. Use an external geocoding API to get coordinates and place a marker on the map.

Phase 5: SRE & Productionization

Goal: Ensure the system is reliable, scalable, and observable, ready for real-world use.

Containerization:
- Write a multi-stage Dockerfile. The first stage builds the Go binary, the second copies the binary into a minimal distroless or alpine base image for a small, secure result.
Configuration Management:
- Move all hardcoded settings (radar lists, thresholds) into a configuration file (e.g., config.yaml) or environment variables. Use a library like Viper to manage this.
Deployment (Docker Compose):
- Write a docker-compose.yml that starts the heloha-server container alongside a Prometheus container scraping its /metrics endpoint.
- Implement proper liveness (/healthz) and readiness probes in your Go server. The readiness check should fail if no radar data has been successfully processed recently.
- Future path: when ready to scale, migrate to Kubernetes manifests (Deployment, Service, Ingress) packaged with Helm. The Dockerfile and /healthz/readiness endpoints you build now translate directly.
Observability:
- Metrics: Add the prometheus/client_golang library. Instrument your code to export key metrics:
  - heloha_radar_files_processed_total{radar="KTLX", status="success"}
  - heloha_data_latency_seconds (time from file timestamp to processing completion)
  - heloha_active_threats_gauge
- Logging: Use Go's stdlib slog with JSON output. Log key events (new threat detected, radar feed lost, tile cache stats).
- Alerting: Configure Prometheus and Alertmanager (included in the Compose stack). Create critical alerts:
  - ALERT RadarFeedDown IF (time() - last_success_timestamp{radar="KTLX"}) > 600
  - ALERT HighProcessingLatency
  - ALERT AppDown (based on container health check)

Backlog / Future Ideas

Mesonet Integration: Add another data ingestion pipeline for the Oklahoma Mesonet. Correlate ground-level wind speed/pressure drops with radar signatures.
Database Persistence: Store historical storm tracks and threat polygons in a time-series or geospatial database (e.g., TimescaleDB, PostGIS).
Push Notifications: Implement a system (e.g., WebSockets or a mobile app) to send real-time alerts to users based on their registered location.
Machine Learning: Train a model on historical radar data and confirmed tornado reports to create a more accurate probabilistic threat model.

13 KiB Raw Blame History