Parallel SITL instances — isolation works, SDF FDM port hardcode is the blocker

TASK-033 closed on the infra side (2026-05-09). Running two full stacks in parallel on a single D2 — the isolation infrastructure works. Full FDM coupling for the second instance is blocked by a separate SDF issue, broken out into a follow-up.

Launch

# instance 0 — defaults
./help_scripts/launch.sh --full --headless -d -log -s sim0

# instance 1 — env override
ENV_FILE=/tmp/.env_sim_inst1 ./help_scripts/launch.sh --full --headless -d -log -s sim1

/tmp/.env_sim_inst1 (matches .env_researchbest on isolation):

SITL_INSTANCE=1
MAVLINK_PORT=5770
GZ_PARTITION=researchbest
ROS_DOMAIN_ID=42
GZ_HEADLESS=1

What works ✅

Layer	Evidence
MAVLink TCP port	`ss -ltn` — 5760 (sim0) and 5770 (sim1), both `LISTEN`, no collision
SITL `-I N` flag	`sim_vehicle.py` got `-I0` / `-I1`, distinct instances
Pre-flight check	first launch passes; second one with the same port would fail (TASK-033 ph2)
`GZ_PARTITION`	sim0 → `partition=sim`, sim1 → `partition=researchbest` (in the gz pane logs)
`ROS_DOMAIN_ID`	sim0 → 0, sim1 → 42; `ROS_DOMAIN_ID=0 ros2 node list` shows only sim0 nodes, `ROS_DOMAIN_ID=42 ros2 node list` shows only sim1
Two Gazebo instances	both `gz sim -s` live in parallel on the RTX 5070 without OOM
ROS 2 nodes per stack	`sensor_monitor` / `sweep` / `autoscan` / `servo_cmd` work independently

What doesn’t work ❌ (separate slice)

Layer	Symptom	Root cause
SITL ↔ Gazebo FDM	sim1 SITL: “Waiting for heartbeat from tcp:127.0.0.1:5770”, `MAV> link 1 down`	`iris_claudedrone/model.sdf:785-786` hardcodes `<fdm_port_in>9002</fdm_port_in>`. Linux `SO_REUSEADDR` allows both binds on `127.0.0.1:9002`, but datagrams reach only one socket → the second SITL gets 0 FDM, never starts the loop

$ ss -lun | grep 9002
UNCONN  960  0  127.0.0.1:9002  0.0.0.0:*
UNCONN  0    0  127.0.0.1:9002  0.0.0.0:*
UNCONN  0    0  127.0.0.1:9002  0.0.0.0:*
UNCONN  0    0  127.0.0.1:9002  0.0.0.0:*

Four binds — both Gazebos + both ArduPilots trying to open the same port. Linux allows it (UDP, SO_REUSEADDR), but physically only one process gets the datagrams; the “lost” SITL waits forever.

Follow-up: SDF instance-aware FDM port

A separate ticket is needed:

Parameterize <fdm_port_in> / <fdm_port_out> in iris_claudedrone/model.sdf via env (e.g., gz sim -s --params-file … or env-templated SDF preprocessing in cmd_gz() in launch.sh).
Default SITL_INSTANCE=0 → 9002/9003 (back-compat).
SITL_INSTANCE=1 → 9012/9013, etc. (+10 * instance, like MAVLink).
ArduPilot side: sim_vehicle.py -I N already shifts its UDP ports by +10 * N — we need the symmetric shift on the Gazebo plugin side.

After that, the acceptance line “both stacks work” closes for real.

HUMAN-MODE startup ritual (phase 7)

Added step 4 to the start ritual in _workspace/agents/simulation/CLAUDE.md:

grep -h "HUMAN-MODE:" _workspace/agents/_shared/HANDOFF*.md 2>/dev/null | tail -2

Behavioral rule: if HUMAN-MODE is active and I’m NOT in the list → start headless + non-default SITL_INSTANCE (≥1), don’t touch port 5760 / GZ_PARTITION=sim. If clear or active with my name → defaults.

The status summary for Aleks now prints a HUMAN-MODE: line.

Sources

~/drone_media/sim/TASK-033/parallel_smoke_evidence.txt — snapshots of ss -ltn / -lun, pgrep, ros2 node list.
~/drone_media/sim/TASK-033/sim{0,1}_{gz,sitl}.log — both stacks.