← gpu-pilot · json · source: CONTINUE_7_17_2026.md

▶ CONTINUE 7/17 (3-GPU national backfill 87% · mobile UX + help tour · OTP-NE systemd · Losing Ground 17 metros · gtfs3 self-heal — newest handoff)

Newest handoff. The national POI-dest backfill (151,781 dests) is 87% done, ~22h to full, driven by THREE GPU producers write-through to the shared Postgres store: geocoder RTX 5080 (unsharded, owns the heavy giant regions = the bottleneck), gtfs3 RTX 5060 Ti (its shard, slow on heavy dests), and a NEW third producer on tfs's idle RTX 2060 (172.26.1.20) scoped via a new PRECOMPUTE_METROS env to the non-giant metros — it cleared the 25.7K small-metro tail in ~3.5h at ~2/s, then expanded to all 43 non-giant metros (DC's 24.6K-stop region peaks 5.26GB, fits the 6GB card, OOM-tested). Also shipped: mobile UX on multimodal.certihomes.com (tap-to-inspect TouchCard + '?' help tour + filled Train/Bus toggles), all 17 new metros live on BOTH losingground-cities and commutingcost-web rankings, OTP-NE revived as a boot-persistent systemd unit on geocoder (Edison→Penn 70/77min via NJT NEC verified), and gtfs3 made self-healing after a power outage (keepalive fd-inheritance + hung-lock fixes). gpu-pilot main=f2ee10f. · updated 2026-07-17 21:04

TL;DR

  • 3-GPU BACKFILL (f2ee10f + tfs setup): geocoder 5080 restarted UNSHARDED (its shard finished) so it sweeps remaining + GPU-computes the giant regions (nyc/socal/midatlantic/la) only it can do; gtfs3 5060 Ti grinds its shard (~0.1/s, weaker card on heavy dests — NOT stalled, GPU 99%, '0.0/s' is log rounding); tfs 2060 (NEW) = 3rd producer on the non-giant tail. Store went 53%→87% over the session. When done: one unsharded straggler pass + restore qwen2.5-coder:14b on gtfs3.
  • tfs WORKER RECIPE: rsync gpu-pilot MINUS the 9 giant region dirs (montreal/chicago/seattle/philly/la/toronto/socal/midatlantic/nec) → engine registers 46 of 55 regions, giants physically absent so it can't OOM on them; venv from gtfs3's exact pip freeze + --extra-index-url=pypi.nvidia.com + ln -sf libcudart.so.12; CUDA_VISIBLE_DEVICES=1 (the 2060, away from GPU-0 image-similarity); producer scoped by PRECOMPUTE_METROS. Small metros compute at ~2/s even on the 2060; big non-giant regions fit 6GB (LRU evicts between them). keepalive+cron with the 9>&- fd-close and timeout -k lessons baked in.
  • MOBILE UX (geo2-live dc18558): isTouch via matchMedia('(hover:none)') → tap a colored hex shows a bottom TouchCard (time/cost + 'Start here'), tap empty map sets start + reverse-geocodes the address; responsive shell (swipeable pill row, collapsible networks/legend, map ≥45vh). Added a floating '?' Help FAB + a 5-step how-it-works tour (auto-shows once via localStorage.mm_help_seen) and FILLED Train/Bus network toggles (were near-invisible thin outlines). Playwright iPhone-13 verified end-to-end.
  • LOSING GROUND / RANKINGS: all 17 new metros now LIVE on BOTH consumers — losingground-cities (geo :3037, 40ae21f) AND commutingcost-web (geo3 :3035, ce2c3a6, which was a stale 29-metro copy the first deploy missed). Added Indianapolis+Louisville this session (re-ranked walk+transit; the zero-transfer guard 86bd8da lets their transit build). walkable 48 / transit 46 metros. GOTCHA: two rankings consumers read the same _50.json — a refresh must deploy to BOTH.
  • OTP-NE ORACLE = systemd unit 'otp-northeast' on geocoder (enabled/boot-persistent, Restart=on-failure), replacing a manual PID. Verified serving: Edison→NYC Penn 70/77min via NJT NEC (matches the validation spec), Times Sq→Penn 23min subway. 14 feeds bundle 40 agencies = all the majors (the '242' was raw source-file count). GOTCHA: JFK AirTrain is NOT in this graph → validate with NEC/subway pairs, not JFK. ETL consumers repointed gtfs3:8091→geocoder:8091 (commit e5802f9). Refresh timer stays disabled pending Krishna's JVM-headroom + NOPASSWD-sudoers calls.
  • gtfs3 SELF-HEAL (after a power outage; UPS since added): root-caused two keepalive bugs — (1) spawned children inherited the flock fd and held it forever (fix: 9>&- closes fd 9 in every child), (2) a hung keepalive ignored SIGTERM and held the lock 12h (fix: cron uses 'timeout -k 15 120' to force-SIGKILL). gtfs3 now self-recovers after reboot to a clean 1/1/1 with the lock free. Same recipe portable to tfs.
  • PENDING: finish backfill (~22h) → unsharded sweep + qwen restore; flip SERVE_STORE_ONLY=1 (store ~87%, close); #79 SKIP_GPU cuda-free serve mode is shipped+validated (3f04c78) but its production flip is gated on the backfill finishing; corridor overlay + res-10 adapter wiring remain from prior handoffs; the every-hex (~1.06M cell) national backfill is a separate bigger job (this run is the 151,781 POI-dest priority slice).

CONTINUE — 2026-07-17 — 3-GPU national backfill (87%), mobile UX, OTP-NE systemd, LG 17 metros, gtfs3 self-heal

Read first: memory national-isochrone-precompute (producer+store+3-GPU+self-heal capstone), mobile-touch-support, losingground-cities-app, interactive-hexmap-and-oracle (OTP-NE), repo-custody-github. Prior handoff: CONTINUE_7_12_2026.md. gpu-pilot main = f2ee10f (pushed via geo bundle). App repos on tlcengine GitHub: multimodal-front-end (geo2-live dc18558), losingground-cities (40ae21f), commutingcost-web (ce2c3a6), commutingcost-api (e5802f9). Engine: geocoder :8098 (systemd hexmap-api, NO --reload → sudo systemctl restart, pw ~/secert.txt). Logs → /var/log/hexmap-api.err.

DONE this session (2026-07-12 → 07-17)

  • ★ 3-GPU national backfill — 87% done (~132.5K/151,781 POI dests), ETA ~22h as of 07-17 evening. Three producers write-through to the shared PG store (commute.iso_precompute, 172.26.1.152:5433):
  • geocoder RTX 5080 — restarted UNSHARDED (its shard 0/2 finished; unsharded sweeps all remaining, store-hits done, GPU-computes the heavy giant regions). ~0.4/s. The completion bottleneck = the ~giant-region dests (nyc/socal/midatlantic/la) only the 5080 can compute.
  • gtfs3 RTX 5060 Ti — shard 1/2, ~0.1/s (slow on heavy dests — weaker card + big regions; NOT stalled, GPU 99%, 0.0/s is log rounding).
  • ★ tfs RTX 2060 (NEW, 172.26.1.20 GPU 1) — 3rd producer for the small→mid-metro tail (see recipe in memory). Cleared the 25.7K small-metro tail in ~3.5h at ~2/s, then user expanded scope to ALL 43 non-giant metros (PRECOMPUTE_METROS env, commit f2ee10f). Big regions fit 6GB (DC 24.6k stops peaked 5.26GB, sfbay 5.30GB — OOM-tested, no OOM). Races geocoder (store dedup).
  • ★ Mobile UX on multimodal.certihomes.com (geo2-live dc18558, prior 24ade88): touch detection (hover:none) → tap hex = bottom TouchCard w/ time/cost + "Start here"; tap empty = set start + reverse-geocode address; responsive shell (swipeable pill row, collapsible networks/legend, map ≥45vh). This session added: "?" Help FAB + 5-step how-it-works tour (auto-shows once via localStorage.mm_help_seen) + filled Train/Bus toggles (were near-invisible thin outlines). Playwright iPhone-13 verified.
  • ★ Losing Ground / rankings — all 17 new metros LIVE on BOTH losingground-cities (geo :3037, 40ae21f) AND commutingcost-web (geo3 :3035, ce2c3a6, was a stale 29-metro copy). Added indy+louisville this session (re-ranked walk+transit; zero-transfer guard 86bd8da lets their transit build). walkable 48 / transit 46 metros. gpu-pilot data 974bd48. TWO rankings consumers must stay in sync (both read data/{most_walkable,best_transit,best_commute}_50.json).
  • ★ OTP-NE oracle = systemd unit on geocoder (otp-northeast, enabled/boot-persistent). Verified serving: Edison→NYC Penn 70/77min via NJT NEC (matches spec). 14 feeds / 40 agencies = all majors (the "242" scare was raw source files). GOTCHA: JFK AirTrain NOT in graph → validate with NEC/subway pairs, not JFK. ETL repoint gtfs3:8091→172.26.1.95:8091 committed (e5802f9).
  • ★ gtfs3 power-outage recovery + SELF-HEAL — power flapped (UPS then added); root-caused a lock-fd inheritance bug (children inherited the keepalive flock and held it forever) + a hung-keepalive-holds-lock-12h bug. Fixed: keepalive.sh closes fd 9 in children (9>&-), cron uses timeout -k 15 120 (force-SIGKILL a hung keepalive). gtfs3 now self-heals after reboot (1/1/1, lock free).
  • SA VIA confirmed already committed+pushed (1aaef77, ancestor of origin/main).
  • commutingcost-api ETL edits committed (e5802f9) — the OTP repoint (was uncommitted on geo3).

PENDING (precise)

  1. Backfill finish (~22h from 07-17 eve). Monitor: tail hexmap-api/precompute.log (geocoder), ssh krish@172.26.1.152 'tail ~/gpu-pilot/hexmap-api/precompute-worker.log' (gtfs3), ssh krish@172.26.1.20 'tail ~/gpu-pilot/hexmap-api/precompute-tfs.log' (tfs). Store %: psql "postgresql://krish@172.26.1.152:5433/commute" -tAc "select count(*) from commute.iso_precompute where engine_rev is not null" / 151781. When done: (a) one final UNSHARDED pass on geocoder to sweep stragglers, (b) ollama run qwen2.5-coder:14b restore on gtfs3, (c) OPTIONAL stop tfs producer+keepalive + hand 2060 back to image-similarity (or leave it for the every-hex backfill).
  2. Flip SERVE_STORE_ONLY=1 (systemd Environment= + restart) once coverage is high — store is ~87% and climbing; safe once hot metros fully in.
  3. #79 CUDA-free serve mode (3f04c78, SKIP_GPU=1) — shipped + validated on :8099; production flip (Environment=SKIP_GPU=1) gated on backfill finishing (a cuda-free :8098 would 503 the producers' compute).
  4. OTP-NE refresh timer (on gtfs3) stays DISABLED pending Krishna: (a) confirm 20g build-JVM fits next to postgres on gtfs3's 31GB, (b) NOPASSWD sudoers on geocoder for systemctl restart otp-northeast (nightly build→serve swap).
  5. Older pending: corridor overlay (cross-metro single-click paint), res-10 adapter wiring (adapter still sends refine=off), every-hex (~1.06M cell) national backfill (separate, bigger job — this run is the 151,781 POI-dest priority slice).

DECISIONS (user chose + why)

  • Add tfs (RTX 2060) as a 3rd producer for the small-metro tail, then expand to all non-giant metros — free idle GPU; giants not rsync'd so it can't OOM.
  • Mobile tap model = tap hex → info card + "Start here"; tap empty → set start (user picked "Info card + Start here" over tap-always-sets-start / long-press).
  • geocoder unsharded (its shard finished) to sweep remaining + own the giants.

GOTCHAS (don't relearn)

  • keepalive MUST use timeout -k not plain timeout N — a hung bash ignores SIGTERM and held the gtfs3 lock 12h. And children MUST get 9>&- or they inherit the flock forever.
  • pkill -f precompute_national self-kills the Bash-tool shell (pattern appears in the command) — bracket-escape precompute_nationa[l].
  • store-count-delta undercounts throughput when producers overlap (re-doing done dests = cheap 40ms store-hits, not +1 rows). Use per-producer ok= for true compute rate.
  • gtfs3 "0.0/s" is log rounding (~0.026/s) — it's the weaker card on heavy regions, NOT stalled (GPU 99%).
  • tfs worker recipe: rsync gpu-pilot MINUS the 9 giant region dirs (montreal/chicago/seattle/philly/la/toronto/socal/midatlantic/nec) → engine registers 46 (55−9), giants can't OOM; venv from gtfs3's exact pip freeze + --extra-index-url=https://pypi.nvidia.com + ln -sf libcudart.so.12 libcudart.so; CUDA_VISIBLE_DEVICES=1 (2060, away from GPU-0 image-similarity); producer scoped via PRECOMPUTE_METROS. RTX 2060 (Turing) works fine with cupy 14.1/RAPIDS 25.4.
  • big regions DO fit the 6GB 2060 (DC 24.6k stops = 5.26GB peak; LRU evicts between big regions so serial is safe).
  • Two rankings consumers (losingground geo:3037 + commutingcost-web geo3:3035) — a ranking refresh must deploy to BOTH.

Comments