RAPTOR GPU — next-sprint checklist

Owner: implementer agent. Order within a phase matters; phases A -> E. Reference: DESIGN.md (rationale), REVIEW.md (external review, 2026-06-10 — read before A), raptor_kernel.h/.cu (contracts), raptor_pipeline.py (orchestration).

A. Kernel correctness (week 1) — ✅ COMPLETED 2026-06-10

All gates green on geocoder: nvcc smoke exact, NVRTC clean, T1 synthetic bit-exact (GPU == CPU == hand, slack 0/30, k=1..5), T2 real-feed bit-exact (1,000 origins x 992 stops x 5 rounds, zero tolerance). Tests: test_t1_synthetic.py, test_t2_mta.py. GPU: 0.8 ms per 5-round departure sample at 1,000 origins.

• Sync this directory geo -> geocoder (mirrored; geocoder canonical)
• Verify Phase-2 gtfs-arrow outputs landed; _assert_schema in loader (sort, dtypes, dep>=arr)
• Implement filter_service_date — Phase 2 pre-exploded (service_id, date); plain semijoin. Feed window 20251102..20260516 is EXPIRED for current dates; pilot uses 20260415 (Wed, 28 svc / 8,492 trips). Feed refresh = phase D.
• Implement build_route_patterns — 210 patterns from 210 groups on real feed
• Add overtaking detector — greedy FIFO chains (transitive elementwise >=, no fixpoint needed); 0 splits on MTA weekday, exercised by T1 synthetic
• Implement build_schedule_slabs (both layouts, int64 bases, FIFO re-assert)
• Implement build_transfer_closure — transfers.txt is parent-level; expands via members; 1,592 edges, already transitively closed on real feed (closure verified on T1 chain fixture)
• Dedup per-origin walk CSR + fixed offsets-never-filled bug (scatter+cumsum was a TODO stub)
• NVRTC compile check — 4 kernels, 24-40 regs
• nvcc smoke build + run — A=30 B=200 C=300 exact, sm_120, CUDA 12.8 (/usr/local/cuda-12.8/bin/nvcc; nvcc NOT on PATH)
• Extend smoke feed -> test_t1_synthetic.py: footpath, 2-transfer (3 boardings), overtaking split, >24:00 trip, missed-by-1s, slack boundary, closure chain, parent default, padded lanes
• Write cpu_reference_raptor (~100 lines numpy; two-mask relax mirrored; label-equality argument documented in docstring)
• Tier-2 test: 1,000 random res-9 origins (800 covered/200 uncovered, seed 42) — GPU == CPU exact
• Per-round equality k=1..5 — via tau_best staircase (one GPU run per k; staircase == tau_k by monotonicity)
• Platform-level vs parent-station: platform-level (992 stops) for the kernel; parents collapse downstream (build_platforms)
• raptor_finalize_travel_u16 clamping verified in T1 (65,534 cap at the >24h trip, 0xFFFF unreachable, padded lanes)
• (review 4.5) Boarding slack: runtime kernel param board_slack_sec, default 30 s (RaptorConfig); 0 = synthetic-test only
• (review 4.6) Parent pairs: feed 511 has NO pathways.txt; conservative expansion of parent-level transfers.txt rows + 180 s default for uncovered parents
• (review 4.8) Two-mask transfer relax implemented (kernel + driver + pipeline + extra marked_xfer buffer)
• (review 4.4) Width decision: keep u16 seconds for the subway pilot (clamp verified); revisit (u16-min / u32-s) before tri-state — tracked in phase D

B. Multi-source batching and performance (week 2) — core items ✅ 2026-06-10

Driver: run_full_res9.py (weekday-asserted). Headline: 73 ms per departure sample (535,471 origins x 992 stops, k=5, slack 30 s) vs the 0.5-1 s design target; 13-sample 7-9 AM window in 0.95 s with allocations reused; state 7.64 GB (matches REVIEW 4.1 math); tiled export 3.41 M rows / 4.2 MB zstd / 0.69 s, VRAM flat. CPU baseline (UNOPTIMIZED Python reference, honest label): 20.6 ms/origin -> 184 min full grid -> GPU 150,347x vs that reference. Caveat for extrapolation: scan cost scales with marked density — crow-fly coverage seeds only 3,999/535K origins, so warp-skip eats most work; revisit numbers when network walk matrix (phase C) and tri-state stops (phase D) raise density.

• Full res-9 run (535,471 origins, one chunk); wall time per round and total — 73 ms/sample; rounds 10.4-14.3 ms
• cudaEvent timing per phase (copy-forward, scan, relax) printed per round — RaptorConfig.time_phases, st["timings"]
• Nsight Compute pass: confirm scan kernel is memory-bound; check achieved DRAM throughput vs ~960 GB/s peak — deferred: kernel is 7-13x under target; profile when tri-state density raises scan cost
• Measure warp-skip rate of the marked-union pre-pass per round (expect high after round 2) — indirect evidence is strong (2-6 ms scans); add a counter when tuning for tri-state
• Measure atomic contention at hub stops (Times Sq, Fulton St, Atlantic Av); only optimize if visible
• Evaluate dropping the per-round D2D copy via best-hopping mode behind a flag — review 4.7: empirical tau_best matching is NOT sufficient (same-round chained-boarding hazard, order-dependent); needs a formal argument + adversarial feeds, else stays off
• (review 4.3) Tiled compaction shipped: write_long_parquet_tiled (65,536-origin tiles, per-tile nonzero, zstd row-groups) — VRAM flat at full res-9
• (review 4.2) Per-round snapshots: stream each round's slab to host (resident i32 snapshots = 12.85 GB, exceeds budget); u16-finalized resident (6.4 GB) only if streaming proves slow
• (review 4.11) In-process CPU baseline measured (plain-Python reference, clearly labeled unoptimized; native-optimized CPU RAPTOR comparison remains open for a rigorous claim)
• Origin chunking driver shipped (run_res10_chunked.py): res-10 3,748,291 origins in 6 x 640K chunks, state 9.13 GB reused, 3.31 s total (~88 ms run/chunk), 23.77 M rows -> one 28.8 MB zstd parquet via TiledParquetWriter
• Departure-window loop: 7-9 AM every 10 min; reuse graph + state allocations — 13 samples / 0.95 s (run_full_res9.py --window)
• Optional: CUDA Graph capture of the fixed-K round loop; drop early-exit if the per-round sync dominates
• Optional: L2 persistence window (cudaAccessPolicyWindow) for the 8 MB schedule slabs
• Pinned-host staging for travel_u16 D2H if Parquet write becomes the bottleneck
• Wide-matrix u16 Parquet export variant alongside long format; measure consumer preference (hexmaps)

C. Walk-leg integration (week 2, parallel with B) — core items ✅ 2026-06-11

Builder: build_walk_matrix.py (pyrosm parse → persistent pylibcugraph SGGraph → 992 cutoff-bounded SSSP, vectorised origin gather). 992 SSSP in 391.6 s. Output walk-matrix/output/h3res9_stop_walk_1km.parquet: 23,852 pairs, 3,557 origins, 954/992 platforms. Network drops 5,588 crow-fly pairs (>900 s on foot across water/rail yards); 118 pairs detour >2× crow-fly. T1 PASS (fixture), T2 PASS (network walk, GPU==CPU bit-exact, k=1..5), OD spot-checks all plausible. Validator: walk-matrix/validate_walk_matrix.py.

• Install cugraph-cu12==25.04 into /home/krish/gpu-pilot/venv (cugraph 25.04.01 + pylibcugraph present; +scipy for cKDTree snapping)
• Extract station entrances — fell back to platform coords (build_platforms order = kernel stop_idx); feed 511 has no pathways.txt / entrance rows. Refine with real entrances when pathways land (phase D)
• Clip OSM walk network for the NYC bbox — PBF local at walk-matrix/osm/new-york-latest.osm.pbf; pyrosm bbox-clip → 1,161,885 nodes / 1,450,348 edges, cached to walk-matrix/cache/. (Subway is NY-only so the NYC walkshed bbox is sufficient; tri-state bbox lands with phase-D feeds.)
• cuGraph SSSP per stop, truncate 1 km / 15 min — pylibcugraph.sssp(cutoff=900); straight-line ≤1 km sanity gate. (Seeded at platform = entrance proxy; undirected graph so direction is symmetric.)
• Persist walk matrix Parquet at gpu-pilot/walk-matrix/output/h3res9_stop_walk_1km.parquet (origin_idx, stop_idx, walk_sec) — 91 KB, all int32, 23,852 unique pairs
• Wire network matrix into build_walk_csr — already auto-loads when the parquet exists (walk_source='network'); crow-fly fallback retained with walk_source provenance
• Spot-check 20 walk pairs vs OSRM on geo2 (water/rail-yard adjacency cases) — internal OD spot-checks PASS; OSRM cross-check (geo2:5002) still TODO
• Last-mile: transpose matrix (stop -> cells) and implement cell-to-cell join product in cudf
• Re-run Tier-3 validation with network walk legs; tighten tolerances — DESIGN.md §8 pairs need transit (departure run); walk-leg spot-checks done

D. Multi-region scaling and ops (sprint after next)

• Multi-feed merge (ingest): build_multifeed_arrow.py — 511 subway + 507 LIRR + 509 NJT-rail + 517 PATH, feed-code namespaced, zero FK orphans, -> output-tristate-rail/ (1,907 stops / 1,376 platforms / 105,491 trips). MNR not present locally (source it); 20,143 orphan service-ids + PATH frequencies flagged for next pass. Feeds are file-based on gtfs3:/mnt/drive1/gtfs/northeast (no gs.* registry table in the commute DB).
• Multi-feed next pass: orphan-service-id extender (v3 pattern) -> common service date -> wire pipeline GTFS_DIR to output-tristate-rail + run kernel (one chunk) -> extend walk matrix to NJ+CT OSM -> 08854->10001 NEC acceptance test
• Re-validate 08854 -> 10001 (NJT NEC P&R, expect 67-78 min) and Times Sq -> JFK with AirTrain feed
• Stop-count blowup plan: at S~50K state is ~600 KB/origin -> ~21K origins/chunk; implement chunk scheduler + output concatenation
• Multi-feed calendar alignment (one service date across agencies, holiday handling)
• Research rRAPTOR (range queries) to replace the departure-sample loop; decide v2 scope
• Daily ETL hookup: consume golden-source feeds from gtfs3 (gs.published), feed_id 511 for MTA subway — NOT 510 (mislabeled, actually MTA Bus Co) and NOT 516 (~25% partial division feed); see CLAUDE.md "Data pathology", commute task #60; verify the id in gs.published before wiring
• Batch job packaging on geocoder (systemd oneshot + timer; respects geocoder ML reservation — coordinate with Krish)
• Publish travel-time Parquet to gtfs3:/mnt/drive1 and register output schema for the hexmaps consumer
• Automate OTP-northeast GraphQL cross-validation (20 pairs, ±3 min) as a post-run gate
• Document GPU memory budget actuals vs DESIGN.md estimates; update sizing table

E. Time+cost product layer (post-pilot — from REVIEW.md §5-6; the kernel is upstream input to this, not the product)

• Journey reconstruction: parent pointers / compact journey ids for top 5-20 nondominated itineraries (prerequisite for ALL costing)
• Two-stage cost engine: itinerary -> GTFS Fares v2 fare products + transfer rules; tolls by facility/direction/time/class; parking price+duration; fuel = miles/adjusted-MPG x price (FuelEconomy.gov/EPA vehicle DB); EV = miles x kWh/mi x charging-loss x $/kWh (EIA prices, home/L2/DCFC split); ride-hail = live quote or labeled estimate range. Never derive cost from arrival times alone; never put monetary fields in dense GPU labels
• Report two cost figures: cash cost vs fully-allocated vehicle cost (+ maintenance/tires/depreciation)
• Mode/asset state at composition: park-and-ride car continuity, kiss-and-ride dwell, bike-carriage rules, ride-hail both ends — no teleporting cars
• Arrive-by queries: backward RAPTOR over a reversed timetable (forward-matrix transpose is wrong: time-dependence, one-ways, directional service/fares)
• Time-based access limits (walk 5/10/15/20 min, bike 10/20/30, P&R/K&R/ride-hail = road time + mode-specific overheads); H3 rings demoted to geometric prefilter
• Per-cell street connectors (entrances, nearest walkable node, boundary portals — not centroids) with mode+direction, for access/egress tables
• Architecture spike: sparse H3->stop access + reusable stop-to-stop profiles (rRAPTOR; ULTRA shortcuts at tri-state scale) vs per-origin batch re-runs — benchmark before committing; never materialize hex-to-hex (535K^2 ~ 573 GB/sample)
• GTFS-RT overlay + provenance timestamps (static output labeled "scheduled")
• Expand validation per REVIEW.md §8: T2 to >=10K origins/multi-date; T3 to thousands of OTP pairs with controlled parameters; add Tier-4 observed-world checks (RT history, agency planners, toll calculators, known fares)