HPC-CARLA

Closed-loop evaluation of autonomous-driving agents as a resilient, schedulable HPC workload. Paper submitted to an SC26 workshop (under review) — J. Kang, S. Jung (2nd author), K. Kim, C. Gardner, J.-S. Yeom, T.-H. Ahn, J.-W. Kim (GIST AIGS / Saint Louis University / LLNL).

This is my working fork. Upstream: jeykang/HPC-CARLA. My contribution: the watchdog + automatic-resubmission job chain that lets multi-hour evaluation jobs survive simulator crashes and allocation expiry without manual intervention.

Campaign results: a persistent CARLA server per GPU (reusing the ~120s boot across jobs) drains a file-locked multi-node queue of agent × route × weather jobs on a 16-GPU (2 × 8 A100) SLURM cluster, collecting per-route CARLA-Leaderboard driving scores. Episode-granular accounting recovered 94% of 13,059 route evaluations from jobs that job-level accounting would have discarded. Six published agents run as declarative YAML pipelines over one audited stage library.

The pipeline is hardened for hostile hardware: the A100 is a compute GPU with a weak real-time rasteriser, so CARLA renders slowly and segfaults intermittently at GL-context creation — the system recovers from that rather than crash. A portable launcher under examples/ reproduces the same run on other clusters (A100 ↔ L40S).

Deep reference: PAPER_REFERENCE.md is the living technical reference — architecture, every agent, the difficulty model, and cluster reliability. This README is the entry point; that file is the detail.

Every agent is a CARLA-Leaderboard method re-implemented as a modular inference pipeline — a YAML list of composable pipeline_modules.py stages driven by consolidated_agent.py, not a monolithic class. Sensors, model, and control are independently configurable without touching agent code, and a generic TorchModelRunner loads any checkpointed model from {module, class_name, args}.

Four of the five productive agents are camera-only; only InterFuser (and LAV) use LiDAR. That modality split drives the difficulty analysis below. Agent weights are gitignored and fetched on demand by each agent's fetch_weights.sh.

.
├── continuous_cli.py                 # single entry point: reset / start / monitor / summary / export
├── manage_continuous.py              # job queue + scheduler (difficulty + illumination coverage)
├── carla_server_manager.py           # per-GPU persistent CARLA server + segfault resilience
├── carla_health_manager.py           # out-of-band server health monitoring
├── continuous_collection_persistent.sh   # SLURM coordinator (one per allocation)
├── persistent_carla_worker.sh        # per-GPU worker (server + agent)
├── leaderboard/team_code/
│   ├── consolidated_agent.py         # universal agent: runs a YAML pipeline each tick
│   ├── pipeline_modules.py           # reusable pipeline stage classes (~2500 lines)
│   ├── configs/{tcp,interfuser,cilrs,neat,roach,lav}.yaml
│   └── {tcp,interfuser,cilrs,neat,roach,lav}/   # per-agent model code + fetch_weights.sh
├── tools/                            # analysis (see below): harvest_results, difficulty_validation,
│                                     #   sensitivity_matrix, weather_axes, verification_report, ...
├── examples/                         # portable cross-cluster launcher + CLUSTER_SETUP.md
├── collection_state/                 # live job queue, completed_jobs.json, metrics
├── dataset/                          # per-route results.json (+ optional per-frame sensor data)
├── PAPER_REFERENCE.md                # the deep technical reference
└── genfig.py                         # figure generation

Each GPU runs one long-lived CARLA server that workers connect to, eliminating the ~120 s per-job startup. On the A100, GL-context creation segfaults intermittently (a host driver/rasteriser issue, not a code bug); carla_server_manager.py makes this recoverable: it kills the stale server, health-checks the reboot (port open and process still alive), retries, and parks a GPU that cannot boot (so it stops burning the queue) while periodically re-attempting. Per-GPU HOME isolation prevents shader-cache contention. See PAPER_REFERENCE.md §9.

Each route file is a suite of many short routes (a _tiny file holds 300–450), run in sequence and checkpointed per route into results.json. On unstable hardware a server often crashes mid-suite, so whole files rarely finish — but every route completed beforehand is saved, even in jobs the queue marks failed. tools/harvest_results.py recovers those per-route scores from every job's checkpoint; in the final campaign 94% of the 13,059 harvested route-evals came from failed jobs. The reportable unit is the route-eval, and per_route_results.csv is the primary artifact.

manage_continuous.py sorts pending jobs by (attempts, coverage_deficit, coverage_count, −difficulty, agent, −est):

• Job-first / agent-interleaved. Difficulty is agent-independent (route + scenario + weather), so all agents' jobs for a (route, weather) tie and the agent field interleaves them — the queue drains as [all agents × condition 1], [all agents × condition 2], …, so an early cutoff leaves every agent with balanced coverage on the same conditions. (This replaced an old hard-coded agent priority that drained one agent first and starved the others.)
• Hardest-first. Within a tier the hardest (route+scenario+weather) runs first, so prune can drop the easier same-route variants as redundant.
• Illumination-stratified coverage. Pure hardest-first collapses the completed sample onto the darkest+rainiest presets (night). COVERAGE_QUOTA (env, default 3) guarantees that many finished jobs per (agent, illumination-bin ∈ {noon, sunset, night}) before reverting to hardest-first; COVERAGE_QUOTA=0 restores the original sort exactly.

Each job gets a scalar difficulty (route geometry + scenario density + weather) used for scheduling and pruning. But a single scalar washes out against performance because different architectures fail on different axes. So difficulty is also analysed as a vector:

• tools/weather_axes.py decomposes the 0–20 weather ordinal into physical axes (illum_dark / precip / road_water / cloud / fog).
• tools/sensitivity_matrix.py fits a per-agent noisy-OR failure model P(fail)=1−exp(−Σ λⱼ·xⱼ), reporting each axis's hazard weight per agent with CIs — and flagging axes the current sample can't identify. Validated on synthetic ground truth (tools/noisy_or_sanity.py): it recovers, e.g., that camera-only agents are illumination-sensitive while LiDAR agents are not.
• tools/difficulty_validation.py correlates difficulty against per-route driving score.

From the login node — the CLI generates and submits the SLURM job (do not sbatch by hand):

# 1. Build the queue (agent × route × weather). --smoke = tiny ~1h validation queue.
python3 continuous_cli.py reset            # or: reset --smoke

# 2. Launch persistent workers on SLURM (one CARLA server + one agent per GPU).
python3 continuous_cli.py --persistent start --slurm \
    --slurm-nodes 1 --slurm-gpus 8 --slurm-time 48:00:00 \
    --slurm-nodelist hpc-pr-a-pod17

# 3. Watch it (the run keeps going if you stop watching).
python3 continuous_cli.py --persistent monitor

The run is resumable (rerun start to reuse completed work) and self-balances across agents.

After / during a run — read the results:

python3 tools/harvest_results.py        # per-route scores (the real metric) + illumination coverage
python3 tools/verification_report.py    # per-agent pass-rate / score, by town/weather
python3 tools/classify_outcomes.py      # agent-result vs infra-failure taxonomy
python3 tools/sensitivity_matrix.py     # per-axis × per-agent difficulty sensitivity

Running on a different cluster (L40S)? See examples/CLUSTER_SETUP.md and the one-file launcher examples/run_cluster.sh — edit one config block, then setup → smoke → run → export.

The full-factorial sweep is complete and drained: 13,059 route evaluations across the five productive agents, with balanced illumination coverage. Key findings reported in the paper: the offline difficulty score turned out to be wrong-signed at scale (pooled Spearman +0.44 across all 13,059, despite an apparent -0.64 at n≈204), and the surviving predictive signal is illumination + map urban density — significant for every evaluated agent. A cross-hardware portability and score-agreement study was run on an L40S cluster.

CILRS scores low by design — audited end-to-end and confirmed genuine weak-baseline behaviour, not an integration bug (PAPER_REFERENCE.md §5.4). Use tools/harvest_results.py to regenerate current numbers from the checkpoints.

• PAPER_REFERENCE.md — full architecture, per-agent detail, difficulty model, reliability.
• examples/CLUSTER_SETUP.md — reproduce on your cluster (A100 ↔ L40S).
• leaderboard/team_code/PIPELINE_MODULES.md — pipeline stage catalogue + context-key contract (to add/modify an agent).