CLAUDE.md — GPU Hex-Precompute Pipeline

Read this first. Self-contained context for the new Claude Code chat scoped to GPU RAPTOR kernel implementation on geocoder.tlcengine.com.

Owner: Krishna Malyala (krish) Host: geocoder.tlcengine.com (172.26.1.95) Working dir: /home/krish/gpu-pilot/ Started: 2026-06-10 Status: Phase 1 (H3 grid) + Phase 2 (GTFS→Arrow) shipped. Phase 3 kernel correctness DONE 2026-06-10 — TODO.md phase A complete: nvcc smoke + NVRTC green, T1 synthetic and T2 real-feed (1,000 origins × 992 stops × k=1..5) bit-exact GPU == CPU, 0.8 ms per 5-round departure sample at 1K origins. Active sprint: TODO.md phase B (full-res-9 scale run, perf instrumentation) + C (cuGraph walk matrix). Gotcha: feed 511 snapshot's calendar expired 2026-05-16 — use service_date=20260415; refresh feed before production (phase D).

What this project is

Replace a CPU OpenTripPlanner polar-sampling precompute (10-day NYC tri-state ETA, repeatedly saturates OTP) with a multi-source RAPTOR kernel running on a single RTX 5080 (16 GB VRAM). Target speed-up: 1000–10,000×. The original iso_precompute_v2.py on gtfs3 hits OTP's 15-thread Grizzly HTTP pool with polar samples that fail A* timeouts on unreachable targets; we're replacing that whole architecture with batched RAPTOR + cuGraph SSSP.

Hardware + environment

Installed in venv: cudf 25.04, cuspatial 25.04, cupy 14.1, h3-py 4.5, pyarrow 19, numba-cuda 0.4, polyline, markdown, pygments.

RAPIDS install path that worked (for re-creating env on a fresh host):

pip install --extra-index-url=https://pypi.nvidia.com \
    'cudf-cu12==25.4.*' 'cuspatial-cu12==25.4.*' cupy-cuda12x
pip install 'h3>=4.0' pyarrow

No conda needed. RAPIDS wheels self-contain CUDA 12 runtime libs even though system CUDA is 13.

Directory tree

/home/krish/gpu-pilot/
├── venv/                              ← RAPIDS 25.4 environment
├── output/
│   ├── nyc_tristate_h3_res9.parquet   ← 535,471 cells, 12 MB
│   └── nyc_tristate_h3_res10.parquet  ← 3,748,291 cells, 86 MB (RETIRED — res-10 not served; see "Map resolution contract")
├── gtfs-arrow/
│   ├── scripts/
│   │   ├── convert_gtfs_to_arrow.py
│   │   ├── validate_gpu_roundtrip.py
│   │   └── benchmark_raptor_shape.py
│   ├── raw/gtfs/                      ← MTA Subway feed 511 raw zip + extracted
│   └── output/
│       ├── stop_times.parquet         ← 2,221,741 rows, sorted by (trip_id, stop_sequence)
│       ├── trips.parquet              ← 76,058 rows
│       ├── routes.parquet             ← 29 rows
│       ├── stops.parquet              ← 1,488 rows
│       ├── service_calendar.parquet   ← 5,061 rows (calendar × dates, exceptions applied)
│       └── encoding_dict.parquet      ← 77,886 string→int32 mappings
├── raptor-design/                     ← Fable 5's design + skeleton
│   ├── DESIGN.md                      ← read first (§11 = accepted review deltas)
│   ├── REVIEW.md                      ← external design review 2026-06-10 (errata verified; read before implementing)
│   ├── TODO.md                        ← sprint plan, phases A-E (E = post-pilot time+cost product layer)
│   ├── raptor_kernel.h                ← 357 lines — structs + kernel signatures
│   ├── raptor_kernel.cu               ← 543 lines — skeleton + host driver + smoke test
│   └── raptor_pipeline.py             ← 480 lines — cupy + cudf orchestration
└── CLAUDE.md                          ← this file

Architecture summary (from Fable 5's design)

• Algorithm: Multi-source RAPTOR (Delling/Pajor/Werneck, "Round-Based Public Transit Routing", Transportation Science). K rounds where K = max_transfers + 1 (typically 4-8). Per round: scan every route touching a marked stop, propagate earliest arrival times.
• Parallelism: One warp per (origin-batch-of-32, route-pattern). Lane = origin index. Stop-major/origin-minor tau[stop][origin] layout so arrival updates from 32 lanes coalesce into a single 128-byte memory transaction. Per-(batch, stop) u32 mark word lets a whole warp skip an unmarked pattern with one scalar load.
• Round semantics: Exact paper. Per-round device-to-device copy-forward (~3 ms) keeps snapshots equal to the (arrival_time, transfer_count) Pareto staircase; CPU-reference validation is bit-exact. Per REVIEW.md: snapshots must be streamed to host per round (resident i32 snapshots = 12.85 GB, over budget); transfer relax uses separate in/out mark buffers; boarding slack is nonzero in v1; "best-hopping" (skip the copy) stays OFF — unproven, same-round chained-boarding hazard.
• Walk integration: Precomputed cuGraph SSSP walk matrix [H3 cell × nearby stops within 1 km], persisted to Parquet, loaded into Round 0 initialization. Crow-fly haversine is the fallback for kernel bring-up before cuGraph install lands. NYC water + rail yards make Euclidean wrong by 10-30 min where you can't afford it.

Known risks (from the design risk register)

1. Service-day handling — MTA WKD/SAT/SUN IDs, calendar_dates.txt exceptions, >24:00:00 times. Mitigation: host-side date filtering plus exact CPU-reference tests. This is the same class of bug as the NJT NEC orphan-service-id issue we fixed in v3 of validate_fix_gtfs.py.
2. Trip overtaking — breaks binary-search monotonicity on stop_times within a route pattern. Mitigation: pattern builder splits by sequence hash, then detects + re-splits overtakers.
3. State blowup at tri-state scale (~50K stops with LIRR + Metro-North + NJT). Mitigation: origin chunking is built into the design (~780K origins/chunk at S=1,000 stops; ~15K at S=50K — corrected per REVIEW.md §4.1: state is 14.25 B per stop-origin, not 16.3 KB/origin).

Performance targets

• Per-origin RAPTOR query: ~1–10 ms (was 302 s on OTP polar)
• Full 535K res-9 origins × one departure window: 0.5–1 s wall
• vs 10-day CPU baseline → 1000× minimum, 10,000× upside with kernel tuning

Map resolution contract (task #45 — audited 2026-06-26, AUTHORITATIVE)

The single served default is H3 res-9. Every live map surface is driven by the engine POST /v1/isochrone path, which emits:

So the "res-9 default, res-8 when coarser is fine" scheme is already in production — res-8 is the natural drive collapse, res-9 is everything else.

res-10 is RETIRED for serving. The ONLY code path that ever served res-10 is the static GET /v1/tiles/{region}/{res}/... route in hexmap-api/app.py, which reads TILES_DIR=/home/krish/gpu-pilot/hexmap-precompute/tiles. That directory does not exist and no manifest is built, so /v1/tiles/* (both res-9 AND res-10) returns 404 in production and is dead for every surface. The live engine never emits res-10. The output/nyc_tristate_h3_res10.parquet (86 MB) and hexmap-tiles/out/**/res10/ are stale Jun-10/Jun-12 experiments, not wired to anything live.

Surface → resolution map (verified live 2026-06-26):

KNOWN DEAD CODE / RECOMMENDATIONS (NOT changed — would require redeploying a live map): - .geo3-map-stage/src/components/HexMap.tsx (deployed as geo3 commutingcost-hexmap) has a zoom-driven tileRes: 'res9'|'res10' selector (RES10_ZOOM=12.2) and a default colorMetric='precomputed'. Both feed the dead /v1/tiles/ path, so: (a) res-10 can NEVER be selected (gated on a manifest that 404s), and (b) the default precomputed choropleth is empty until the user clicks (which flips to the working live-isochrone layer). The map still works interactively; this is latent, not user-blocking. Recommended (separate task, needs rebuild+redeploy of commutingcost-hexmap on geo3): delete the tileRes/RES10_ZOOM/loadManifest/loadTile machinery and default the map to the live isochrone (or build hexmap-precompute/tiles if static tiles are wanted). Left as-is here to avoid degrading the live map per task #45's no-break rule. - If res-10 serving is ever revived, build hexmap-precompute/build_tiles.py → TILES_DIR first; the app route already supports res in {res9,res10}.

Sprint plan (lift directly from raptor-design/TODO.md)

Phase 1 (was Fable 5's design output, done): Architecture + skeleton. Phase 2 (now): Kernel correctness on toy input. Get the smoke test in raptor_kernel.cu running end-to-end on a 3-trip 5-stop hand-built fixture; validate bit-exact against a CPU-side Python RAPTOR. Phase 3: Multi-source batching. Scale to 32-origin warps, then 1024-origin blocks. Validate VRAM stays under 12 GB on res-9 (535K origins). Phase 4: Walk-leg integration. Wire in cuGraph SSSP for OSM street network → station entrances. Validate against 5 hand-picked OD pairs (see Validation OD pairs below). Phase 5 (later): Multi-region scaling. Add LIRR + Metro-North + NJT feeds (still <200 MB total). Then SE, Midwest, MW, SC.

Validation OD pairs (ground truth)

CPU reference: query http://gtfs3.tlcengine.com:8091/otp/gtfs/v1 (when serving) with the same OD + date + time; compare arrival times within ±2 minutes.

Critical constraints — DO NOT

• DO NOT touch port 8030 on this host. Second Brain Parser API runs there; it holds 2 GB GPU VRAM and indexes 282+ sessions / 156K+ messages. Leave it alone.
• DO NOT install RAPIDS into /home/krish/api-env — that's the Second Brain venv. Use /home/krish/gpu-pilot/venv only.
• DO NOT run the kernel on geo or geo2 — they're off-limits for the hexmap project per global CLAUDE.md (different production workloads). RTX 5080 on geocoder is the right host.
• DO NOT use the Zhou-Zeng GA* algorithm for transit routing. That paper (AAAI 2015, 45× speedup) is for general A on combinatorial search (Rubik's cube etc.). RAPTOR has no priority queue, no heuristic, no node-duplication detection — those are the things that make A hard on SIMD. RAPTOR is naturally GPU-friendly via bounded rounds + atomicMin on i32 arrival arrays.

Living report + build log (keep updated!)

The design docs + sources + a running build log render to https://videolens.certihomes.com/r/2026-06-10-raptor-design-review.html (videolens Next.js app on geo:172.26.1.45, file at geo:~/videolens/data/reports/). At every milestone: append a dated entry at the TOP of raptor-design/SESSION_LOG.md, then run venv/bin/python raptor-design/publish_report.py (renders with raptor-design/report_theme/, scp's to geo — geocoder's key is authorized).

Reference reading order

1. /home/krish/gpu-pilot/raptor-design/DESIGN.md — Fable 5 architecture doc (§11 = review deltas)
2. /home/krish/gpu-pilot/raptor-design/REVIEW.md — external review: verified errata (memory math, u16 clamp, cp.nonzero OOM), pre-mortem, 5-tier test plan, time+cost product gaps
3. /home/krish/gpu-pilot/raptor-design/TODO.md — sprint plan
4. /home/krish/gpu-pilot/raptor-design/raptor_kernel.h — types + signatures
5. /home/krish/gpu-pilot/raptor-design/raptor_kernel.cu — skeleton + smoke test
6. Delling/Pajor/Werneck 2015 "Round-Based Public Transit Routing" — the algorithm
7. /home/krish/commute/CONTINUE.md v6 on geo — broader project context (transit fleet, OTP issues, regional graphs)
8. Veritasium 2026-05-30 "Google Maps is unreasonably fast" (video-lens report on file) — why preprocessing matters

Data pathology to remember

The Phase 2 agent surfaced a real gs.feeds bug: feed 510 is mislabeled "MTA Subway" but is actually MTA Bus Company. 511 is the canonical MTA NYCT (subway + commuter rail) feed — this is what gtfs-arrow/output/ was built from. Feed 516 has ~25% of subway trips (partial division feed). Tracked as task #60 in the commute repo.

The v3 validate_fix_gtfs.py orphan-service-id extender (NJT NEC bug pattern) applies to multiple US transit systems: Phoenix Valley Metro, Salt Lake UTA, Denver RTD, ABQ RIDE, Tucson SunTran, etc. Same fix can flow through the multi-feed union step when we extend Phase 2 beyond MTA Subway.

Off-host things to know about (don't depend on them being available)

• OTP-NE at http://gtfs3:8091/otp/gtfs/v1 — useful as a CPU reference for transit queries; may be intermittent under hex-precompute load (see task #54)
• OSRM (US) at http://172.26.1.151:5002 — US drive routing (us-latest, MLD, car.lua). Border-snaps Canadian points ~30-50 km to the nearest US node — DO NOT use it for the 6 CA metros.
• OSRM (Canada) at http://172.26.1.151:5004 — Canada drive routing (canada-latest 2026-06-23, MLD, car.lua; docker osrm-canada, restart=unless-stopped, data in /home/krish/osrm-canada). Built 2026-06-25 to fix CA border-snap. Toronto/Vancouver/Montreal/Calgary/Edmonton/Ottawa interior hops verified sane (3-5 min) vs :5002 garbage (0 min, 50 km snap). Engine wiring: route the 6 CA metros' drive/uber/PnR/KnR legs here. Single source of truth = raptor-design/multimodal.py:37 OSRM_BASE; add an osrm_base_for(lat,lon) country router that returns :5004 for CA points (reuse metro_for() / _METRO_BBOX which already has the 6 CA boxes).
• OSRM (foot) at http://172.26.1.151:5003 — walk profile (nymetro); OSRM_FOOT_BASE
• Postgres at gtfs3:5433 (commute DB, user krish trust-auth from internal IPs) — gs.feeds, gs.agency, gs.routes, gs.stops for the Golden Source registry
• API at https://api.commutingcost.com/v1/trip/plan — production trip planner; this kernel will eventually replace its OTP transit backend

When you start: source venv/bin/activate, read raptor-design/DESIGN.md, then begin Phase 2 of the sprint plan (kernel correctness on toy input).

CPU OTP reference endpoints (for GPU validation)

The CPU OTP graphs are now reserved purely as validation oracles for the GPU RAPTOR kernel. Do NOT use them in production hot paths.

When validating GPU RAPTOR output against these: - ±2-3 min tolerance per leg (OTP routing isn't deterministic beyond pattern selection) - Pin date to a non-holiday weekday in service window (2020-01-01 to 2050-12-31) - Match transitModelTimeZone: NE = America/New_York, MW = America/Denver, SC = America/Chicago, SE = America/New_York

MW data quality fixes applied (worth knowing for GPU MW expansion): - 27 feeds had non-monotonic block_ids (v8 logged) — RTD-2054 worst at 18,814 trips - 92 of 156 feeds had GTFS-flex auxiliary files stripped (areas.txt, stop_areas.txt, locations.geojson, location_groups.txt removed; start_service_area_id + end_service_area_id columns nulled in stop_times.txt) - 3 RTD feeds (178, 2054, 2280) all present in build; gs.feeds dedup pending - 0 InterliningTeleport in clean build (was 41K before patches)

OTP 2.7.0 PR #6211 fixed the Raptor NPE in TripFrequencyAlightSearch that was blocking 4 of 5 smoke tests on OTP 2.6.0. All GPU MW validation should use OTP 2.7.0+ on the gtfs1 reference.