{
  "project": "gpu-pilot",
  "doc": "CONTINUE_7_17_2026.md",
  "title": "\u25b6 CONTINUE 7/17 (3-GPU national backfill 87% \u00b7 mobile UX + help tour \u00b7 OTP-NE systemd \u00b7 Losing Ground 17 metros \u00b7 gtfs3 self-heal \u2014 newest handoff)",
  "oneline": "Newest handoff. The national POI-dest backfill (151,781 dests) is 87% done, ~22h to full, driven by THREE GPU producers write-through to the shared Postgres store: geocoder RTX 5080 (unsharded, owns the heavy giant regions = the bottleneck), gtfs3 RTX 5060 Ti (its shard, slow on heavy dests), and a NEW third producer on tfs's idle RTX 2060 (172.26.1.20) scoped via a new PRECOMPUTE_METROS env to the non-giant metros \u2014 it cleared the 25.7K small-metro tail in ~3.5h at ~2/s, then expanded to all 43 non-giant metros (DC's 24.6K-stop region peaks 5.26GB, fits the 6GB card, OOM-tested). Also shipped: mobile UX on multimodal.certihomes.com (tap-to-inspect TouchCard + '?' help tour + filled Train/Bus toggles), all 17 new metros live on BOTH losingground-cities and commutingcost-web rankings, OTP-NE revived as a boot-persistent systemd unit on geocoder (Edison\u2192Penn 70/77min via NJT NEC verified), and gtfs3 made self-healing after a power outage (keepalive fd-inheritance + hung-lock fixes). gpu-pilot main=f2ee10f.",
  "tldr": [
    "3-GPU BACKFILL (f2ee10f + tfs setup): geocoder 5080 restarted UNSHARDED (its shard finished) so it sweeps remaining + GPU-computes the giant regions (nyc/socal/midatlantic/la) only it can do; gtfs3 5060 Ti grinds its shard (~0.1/s, weaker card on heavy dests \u2014 NOT stalled, GPU 99%, '0.0/s' is log rounding); tfs 2060 (NEW) = 3rd producer on the non-giant tail. Store went 53%\u219287% over the session. When done: one unsharded straggler pass + restore qwen2.5-coder:14b on gtfs3.",
    "tfs WORKER RECIPE: rsync gpu-pilot MINUS the 9 giant region dirs (montreal/chicago/seattle/philly/la/toronto/socal/midatlantic/nec) \u2192 engine registers 46 of 55 regions, giants physically absent so it can't OOM on them; venv from gtfs3's exact pip freeze + --extra-index-url=pypi.nvidia.com + ln -sf libcudart.so.12; CUDA_VISIBLE_DEVICES=1 (the 2060, away from GPU-0 image-similarity); producer scoped by PRECOMPUTE_METROS. Small metros compute at ~2/s even on the 2060; big non-giant regions fit 6GB (LRU evicts between them). keepalive+cron with the 9>&- fd-close and timeout -k lessons baked in.",
    "MOBILE UX (geo2-live dc18558): isTouch via matchMedia('(hover:none)') \u2192 tap a colored hex shows a bottom TouchCard (time/cost + 'Start here'), tap empty map sets start + reverse-geocodes the address; responsive shell (swipeable pill row, collapsible networks/legend, map \u226545vh). Added a floating '?' Help FAB + a 5-step how-it-works tour (auto-shows once via localStorage.mm_help_seen) and FILLED Train/Bus network toggles (were near-invisible thin outlines). Playwright iPhone-13 verified end-to-end.",
    "LOSING GROUND / RANKINGS: all 17 new metros now LIVE on BOTH consumers \u2014 losingground-cities (geo :3037, 40ae21f) AND commutingcost-web (geo3 :3035, ce2c3a6, which was a stale 29-metro copy the first deploy missed). Added Indianapolis+Louisville this session (re-ranked walk+transit; the zero-transfer guard 86bd8da lets their transit build). walkable 48 / transit 46 metros. GOTCHA: two rankings consumers read the same _50.json \u2014 a refresh must deploy to BOTH.",
    "OTP-NE ORACLE = systemd unit 'otp-northeast' on geocoder (enabled/boot-persistent, Restart=on-failure), replacing a manual PID. Verified serving: Edison\u2192NYC Penn 70/77min via NJT NEC (matches the validation spec), Times Sq\u2192Penn 23min subway. 14 feeds bundle 40 agencies = all the majors (the '242' was raw source-file count). GOTCHA: JFK AirTrain is NOT in this graph \u2192 validate with NEC/subway pairs, not JFK. ETL consumers repointed gtfs3:8091\u2192geocoder:8091 (commit e5802f9). Refresh timer stays disabled pending Krishna's JVM-headroom + NOPASSWD-sudoers calls.",
    "gtfs3 SELF-HEAL (after a power outage; UPS since added): root-caused two keepalive bugs \u2014 (1) spawned children inherited the flock fd and held it forever (fix: 9>&- closes fd 9 in every child), (2) a hung keepalive ignored SIGTERM and held the lock 12h (fix: cron uses 'timeout -k 15 120' to force-SIGKILL). gtfs3 now self-recovers after reboot to a clean 1/1/1 with the lock free. Same recipe portable to tfs.",
    "PENDING: finish backfill (~22h) \u2192 unsharded sweep + qwen restore; flip SERVE_STORE_ONLY=1 (store ~87%, close); #79 SKIP_GPU cuda-free serve mode is shipped+validated (3f04c78) but its production flip is gated on the backfill finishing; corridor overlay + res-10 adapter wiring remain from prior handoffs; the every-hex (~1.06M cell) national backfill is a separate bigger job (this run is the 151,781 POI-dest priority slice)."
  ],
  "updated": "2026-07-17 21:04",
  "sections": [
    {
      "level": 1,
      "title": "CONTINUE \u2014 2026-07-17 \u2014 3-GPU national backfill (87%), mobile UX, OTP-NE systemd, LG 17 metros, gtfs3 self-heal"
    },
    {
      "level": 2,
      "title": "DONE this session (2026-07-12 \u2192 07-17)"
    },
    {
      "level": 2,
      "title": "PENDING (precise)"
    },
    {
      "level": 2,
      "title": "DECISIONS (user chose + why)"
    },
    {
      "level": 2,
      "title": "GOTCHAS (don't relearn)"
    }
  ],
  "stats": {
    "lines": 41,
    "words": 949
  },
  "html": "https://board.certihomes.com/project-docs/gpu-pilot/CONTINUE_7_17_2026.html",
  "out_name": "CONTINUE_7_17_2026.md",
  "comments_api": "https://board.certihomes.com/api/doc-comments?project=gpu-pilot&doc=CONTINUE_7_17_2026.md"
}