← gpu-pilot · json · source: CONTINUE_7_12_2026.md

▶ CONTINUE 7/12 (Serve-from-Postgres · rail-nec 34× · metro priority-2 up to 122× · DUAL-GPU sharded producers · repo custody → tlcengine · fares seed — newest handoff)

Newest handoff. THE architecture flip: GPU is now an ETL PRODUCER only — production serves disk(L1) → Postgres commute.iso_precompute (L2) → GPU(L3 fallback), rows stamped ENGINE_REV so stale never serves; SERVE_STORE_ONLY=1 removes the GPU from interactive serving entirely. The national backfill went ~18 days → ~8-9: rail-focused nec (78K→5.4K stops, corridor cell 94s→2.8s, Edison→Philly still 75min via NJT→Amtrak Keystone — proven per-leg by trace_trip.py) + priority-2 metros (Baltimore 122×, SD 74×, Philly 18×, DC 11×, LA 2.7×) + a SECOND sharded producer on gtfs3's RTX 5060 Ti (PRECOMPUTE_SHARD=k/m, shared PG store). Also: first-visit warming toast on multimodal.certihomes.com, fare-gap audit+seed (20 metros curated, 8 real $0-placeholder rows fixed, 14 majors seeded), and the repo-custody sweep — every deployed app now on tlcengine GitHub (FE was Bitbucket, adapter/crm-admin had NO git; wildcard TLS key kept out of every pushed tree). gpu-pilot main=5622f9b. · updated 2026-07-12 04:53

TL;DR

  • SERVE-FROM-POSTGRES (5c9f797): app.py's isochrone handlers now read disk L1 → PG L2 → GPU L3. _store_get/_store_put keyed by the exact disk-cache stem; ENGINE_REV stamps rows and reads filter on it (invalidate = bump the rev, no wipe gap); write-through on every GPU compute; PG errors fall through to disk+GPU so the live path can't break. Validated: evict disk → PG serve in 36ms, byte-identical, zero GPU. refresh_from_golden.sh = the ETL trigger to wire into the monthly golden cron.
  • RAIL-NEC (0aeae99): the corridor region kept ALL rail (subway/rail/LR/ferry, 219 routes) and dropped ~73K local-bus stops across 5 states → 78,073→5,400 stops, corridor-cell isochrone 94s→2.8s (34×). Edison→Philly 75min intact; trace_trip.py (76d780b, a CPU RAPTOR with parent pointers the GPU kernel lacks) proves the itinerary: NJT NEC Edison→New Brunswick 4min, 136s walk to the Amtrak platform, Keystone Service→Philadelphia. build_rail_nec.sh is self-validating with auto-rollback (backup in output/networks-backups/ — NOT inside output/networks/, where the engine registry would discover it as a region).
  • METRO PRIORITY-2 (37202fd): la/philly/baltimore/dc/san_diego were served by the prio-1 corridor megas (socal 46K / midatlantic 67K stops) at 29-55s despite having their own regions. At priority 2: Baltimore 0.45s (122×), San Diego 0.39s (74×), Philly 3.0s (18×), DC 5.2s (11×), LA 10.9s (2.7×); Times Sq pinned byte-identical. Tradeoff (accepted, reversible per-metro): cross-metro single-click painting (Baltimore↔DC MARC shed, Philly→Wilmington, LA↔SD) deferred to a corridor overlay. Stale mega-computed cells invalidated across disk+PG+progress.
  • DUAL-GPU PRODUCERS (5622f9b): PRECOMPUTE_SHARD=k/m deterministically splits the 151K national dests — geocoder RTX 5080 = shard 0/2, gtfs3 RTX 5060 Ti = shard 1/2 (~72.5K each), both write-through to the SAME Postgres store at the same ENGINE_REV. gtfs3 worker recipe: qwen2.5-coder:14b paused (Krishna-authorized, alpha), ~20GB artifacts rsynced, /home/krish/gpu-pilot symlink so hardcoded paths resolve, pip nvidia-*-cu12 wheels + LD_LIBRARY_PATH (driver-only box, no system CUDA), cache-janitor for the 137GB disk. Validated: Baltimore on the 5060 Ti = identical 4,979 cells. Finish with one UNSHARDED pass, then ollama run qwen2.5-coder:14b.
  • TWO-CLASS KERNELGATE (9bf05f2) + WARMING TOAST: interactive clicks always outrank batch:true producers (Times Sq was 30s/2-cell under producer starvation → 1.26s/15,203). Multimodal's default click resolves to the outbound-transit key, NOT the legacy key the national sweep warms — warm_nyc_multimodal.py filled the real key for 6,421 NYC cells. geo2 FE (pm2 tlccommute) now shows 'First look at this area — computing it now (~30s)' when a click runs >3s; instant cached clicks never see it.
  • FARE-GAP AUDIT + SEED (5622f9b): 415/935 golden feeds have no GTFS fare rows and the top-20 are MAJORS (CTA/TTC/STM/TransLink/WMATA-bus/RTD/MARTA/Miami/Calgary/PACE/OC Transpo...) — big agencies omit fare files. Engine curation already covered most; added 20 metros to CURATED_REGION_FARE (baltimore 2.00 … richmond 0.00 GRTC fare-free … quebec_city CAD 3.75), fixed 8 real systems' $0.00 placeholder base_fare rows in public.transit_fares (they poisoned the geo3 api), seeded 14 major agencies with mdb-number-keyed rows. Remaining $0 rows are genuinely-free circulators/campus shuttles.
  • REPO CUSTODY (memory repo-custody-github): every deployed app now on tlcengine GitHub — multimodal FE was on Bitbucket (vendor history has >100MB blobs → clean snapshot branch geo2-live set as default), /click adapter had NO remote (full history pushed), onboarding was on a contractor's personal account, crm-admin had NO git. The wildcard *.tlcengine.com private key found in mm-api certs/ was verified absent from every pushed tree and is now gitignored. Convention: geo2-live/geo3-live snapshot branches + gh repo edit --default-branch; never force-push over unseen history.
  • POSITIONING (Krishna): sell the HEXMAP DATA — processed per-hex time+cost intelligence — not the golden GTFS corpus ('the jewelry is worth more than the gold bar'); also sidesteps feed-redistribution licensing entirely. Golden corpus stays the internal moat.

CONTINUE — 2026-07-12 — Serve-from-Postgres + rail-nec + priority-2 metros + dual-GPU producers + repo custody + fares seed

Read first: memory national-isochrone-precompute (the producer+store capstone), repo-custody-github, golden-short-calendar-fix, the-pitch-positioning. Prior handoff: CONTINUE_7_8_2026.md. Repos: gpu-pilot main = 5622f9b (pushed via geo bundle). NEW on tlcengine GitHub: multimodal-front-end, multimodal-adapter, multimodaltransit-api, TLCengine-onboarding, crm-admin (see custody memory). Engine: geocoder :8098 (systemd hexmap-api, NO --reload → sudo systemctl restart, pw ~/secert.txt). Logs → /var/log/hexmap-api.err, NOT journalctl.

DONE this session (2026-07-09 → 07-12)

  • ★ MOBILE SWEEP (post-wrap 07-12) — audited all 5 public surfaces on iPhone emulation; two were broken, both fixed + deployed. multimodal.certihomes.com was unusable on phones: hover-only hex/station tooltips (every tap moved the pin) AND the map sat entirely below the fold (tagline + wrapped pills + agency-legend ate 3+ screens). Fix: isTouch via matchMedia('(hover: none)') → tap-on-feature shows a bottom TouchCard ("Start here" + ✕), empty-tap dismisses-then-moves-pin; responsive shell (single swipeable pill row, "Networks & legend" collapse, map ≥45vh). FE snapshot 24ade88 (tlcengine/multimodal-front-end geo2-live). commutingcost.com/map: left control panel covered the whole phone → mobile "⚙ Controls" collapse, geo3 188fad8 + stage mirror fc25738. Desktop verified unchanged (hover tip, click-move-pin, panel visible, no toggle). cc_home/losingground/TLCcosts mobile OK. Gotchas + deploy recipes in memory mobile-touch-support.

  • KernelGate wedge fixed (9697433) — a stranded permit 429'd every cold isochrone for 14h while the GPU sat idle; restart clears it. Then two-class gate (9bf05f2): interactive clicks ALWAYS outrank batch:true producers (Times Sq click went 30s/2-cell → 1.26s/15,203). Multimodal's default click = outbound-transit key ≠ the legacy key the national sweep warms → warm_nyc_multimodal.py (6,421 NYC cells, done).

  • ★ SERVE-FROM-POSTGRES (5c9f797) — Krishna's architecture call: GPU = ETL producer only; production reads disk L1 → PG commute.iso_precompute L2 → GPU L3 fallback. ENGINE_REV stamps rows (stale never served; invalidate = bump rev), SERVE_STORE_ONLY=1 env removes GPU from interactive serving, refresh_from_golden.sh = the ETL trigger. Validated: evict disk → PG serve 36ms, byte-identical, no GPU.
  • ★ Rail-focused nec (0aeae99) — nec 78,073 → 5,400 stops (dropped 5-state local bus; kept ALL rail: subway 2,059 / rail 1,657 / LR 628 / ferry 41 stops). Corridor cell 94s → 2.8s (34×). Edison→Philly 75min intact — trace_trip.py (76d780b) proves the itinerary: NJT NEC Edison→New Brunswick, 136s walk, Amtrak Keystone→Philly (the engine picks Amtrak by itself; Metropark is northbound = wrong way). Self-validating builder build_rail_nec.sh (auto-rollback; backup at output/networks-backups/nec.bak-fullbus-20260709).
  • ★ Metro priority-2 (37202fd) — la/philly/baltimore/dc/san_diego were dragged through the prio-1 corridor megas (socal/midatlantic) at 29-55s despite having own regions. Bumped to prio 2: Baltimore 0.45s (122×), SD 0.39s (74×), Philly 3.0s (18×), DC 5.2s (11×), LA 10.9s (2.7×); Times Sq byte-identical. Stale mega-computed cells invalidated (disk 4,913 / PG 5,984 / progress 4,975).
  • ★ DUAL-GPU sharded producers (5622f9b + gtfs3 setup) — solo ETA was ~18d. gtfs3 worker (~/gpu-pilot-worker, RTX 5060 Ti, qwen2.5-coder:14b PAUSED with Krishna's OK): symlink /home/krish/gpu-pilot → worker copy (hardcoded paths resolve unchanged), pip NVIDIA runtime wheels + LD_LIBRARY_PATH (no system CUDA there), run_worker_engine.sh (uvicorn 127.0.0.1:8098), cache-janitor (137GB disk). Validated: Baltimore on the 5060 Ti = identical 4,979 cells. PRECOMPUTE_SHARD=k/m env: geocoder=0/2, gtfs3=1/2, ~72.5K dests each, both write-through to the shared PG store at the same ENGINE_REV. Run a final UNSHARDED pass to sweep stragglers.
  • Fare-gap audit + seed (5622f9b) — corrected claim: 415/935 golden feeds fare-less and the top-20 are MAJORS (CTA/TTC/STM/TransLink/WMATA-bus/RTD/MARTA/Miami/Calgary/PACE/OC Transpo/Metro Transit MSP...). Engine already curated most majors; seeded +20 metros into CURATED_REGION_FARE (baltimore 2.00 … richmond 0.00 GRTC fare-free … quebec_city CAD 3.75), fixed 8 real systems' $0.00 placeholder rows in public.transit_fares (poisoned geo3 api), seeded 14 major DB rows (mdb-number-keyed). ~90 remaining $0 rows are genuinely-free circulators/campus shuttles.
  • First-visit warming toast (geo2 FE, deployed) — Map.js: click >3s → pill "First look at this area — computing it now (~30s). It's instant after this." Built + pm2 restart tlccommute; verified in the served bundle. Backup components/Map.js.bak.warmtoast.20260710.
  • Repo custody sweep — everything on tlcengine GitHub (memory repo-custody-github has the full table): FE was on Bitbucket (vendor history has >100MB blobs → pushed clean snapshot branch geo2-live, set default), adapter had NO remote (full history pushed), onboarding was on a contractor's personal gh, crm-admin had NO git. Wildcard *.tlcengine.com private key found in mm-api certs/ — verified 0 key files in any pushed tree, now gitignored.
  • Positioning recorded: sell the hexmap data (processed intelligence), not the golden GTFS ("jewelry vs gold bar") — also sidesteps feed-redistribution licensing. In the-pitch-positioning memory.

PENDING (precise)

  1. Producers running — monitor tail hexmap-api/precompute.log (geocoder shard 0/2) + ssh krish@172.26.1.152 'tail ~/gpu-pilot/hexmap-api/precompute-worker.log' (gtfs3 shard 1/2). Expect ~8-9 days full, hot metros within days. When both finish: one UNSHARDED pass (no PRECOMPUTE_SHARD env) to sweep stragglers, then ollama run qwen2.5-coder:14b to restore the LOS AI on gtfs3.
  2. Flip SERVE_STORE_ONLY=1 when store coverage is high (add Environment=SERVE_STORE_ONLY=1 to the systemd unit + restart). Check coverage: select count(*) from commute.iso_precompute where engine_rev is not null.
  3. #79 Level-2: CUDA-free serve mode — engine flag to skip GPU network load entirely (serve PG/disk only, 503 on miss); frees all VRAM. Build on scratch :8099 first.
  4. Corridor overlay — restore cross-metro single-click painting (Manhattan→Philly, Baltimore↔DC MARC shed, Philly→Wilmington, LA↔SD) lost to the priority scheme: precompute corridor sheds per rail station, overlay at serve time. Alternative for Baltimore↔DC: merged dc+baltimore region rebuild.
  5. res-10 adapter wiring — engine supports refine=auto; geo2 adapter still sends off.
  6. #69 Losing Ground narrative refresh (reports + losingground-cities app on geo :3037).
  7. MobilityData contribution (optional GTM): upstream the golden fixer + QA reports; commercial licensing = hexmap data only.
  8. FE full-history migration (optional): git filter-repo the Bitbucket history minus >100MB blobs → replace the snapshot branch.

DECISIONS (user chose + why)

  • GPU = producer; production reads Postgres — "not dependent on GPU, free for other tasks". Store-only flip gated on coverage.
  • Rail-focused nec over full-bus (corridor = rail's job; local-bus-only hexes in the corridor lose reach — accepted).
  • Priority-2 metros — metro-internal speed over cross-metro paint (Baltimore↔DC etc. deferred to overlay; reversible per-metro).
  • Alpha posture: everything deployable without user-facing risk gates ("everything is not live used by user, we are in Alpha"); qwen on gtfs3 stoppable for compute.
  • Sell hexmap data, not golden GTFS (jewelry > gold bar).
  • All code on tlcengine GitHub — no Bitbucket, no contractor accounts, no unversioned deploys.

GOTCHAS (don't relearn — most also in memory files)

  • hexmap-api logs → /var/log/hexmap-api.err (unit uses StandardError=append:), NOT journalctl.
  • Ruff hook strips imports added before first use → silent NameError swallowed by broad except (the gzip/store bug). Add import + use in the SAME edit.
  • Disk-cache hit returns BEFORE engine.ready → can't probe readiness with a cached point; validation queries must cache-bust + retry on 503. healthz 200 ≠ networks loaded.
  • A backup dir inside output/networks/ registers as a REGION (engine scans */manifest.json). Backups → output/networks-backups/.
  • Region rebuild that changes output needs invalidation across BOTH disk L1 and PG L2 (rev filter only guards PG; disk has no rev). Also drop producer-progress lines for re-warm.
  • pkill -f self-kills the Bash-tool shell if the pattern text appears anywhere in the same command — run pkill isolated + bracket-escape (precompute_nationa[l]).
  • GitHub 100MB hard limit → vendor FE history unpushable; snapshot-branch convention (geo2-live/geo3-live, gh repo edit --default-branch). Auto-mode blocks git push -f.
  • gtfs3 worker recipe: python3.12-venv apt pkg; RAPIDS wheels DON'T bring CUDA runtime (driver-only box) → pip nvidia-*-cu12 wheels + LD_LIBRARY_PATH + ln -sf libcudart.so.12 libcudart.so; numba warning benign (kernel = cupy NVRTC).
  • cat >> .gitignore onto a no-trailing-newline file glues the first line — check git check-ignore after.
  • transit_fares had $0.00 placeholder rows for real agencies — engine curated layer masked it; direct DB consumers weren't. Remaining $0 rows are genuinely free services.

Comments