MAVProxy 'Waiting for heartbeat' — three causes, layered debug

One symptom — MAVProxy: Waiting for heartbeat from tcp:127.0.0.1:5760 forever — turned out to be three independent causes at once. TASK-021 21a closed after fixing all three. The sequential-debug history is useful in itself as an example of layered debugging.

Symptom

RiTW: Starting ArduCopter : .../arducopter --model JSON --speedup 1 --slave 0 \
      --defaults .../copter.parm,.../gazebo-iris.parm,.../indoor.parm \
      --sim-address=127.0.0.1 -I0
Connect tcp:127.0.0.1:5760 source_system=255
Loaded module console
Telemetry log: mav.tlog
Waiting for heartbeat from tcp:127.0.0.1:5760
MAV>                                            ← frozen forever

With --mavros: FCU: Connection problem.

Cause 1 — Orphan UDP 9002 race (dev-log/04)

I checked step by step: arducopter binary alive, param files readable, sim_vehicle.py itself OK, TCP 5760 in LISTEN. Looked at the ArduCopter window:

No JSON sensor message received, resending servos
No JSON sensor message received, resending servos
... (forever)

arducopter --model JSON runs in lock-step: ArduPilot sends servo commands to UDP 127.0.0.1:9002 (the plugin listens), the plugin sends IMU/baro/GPS back — ArduPilot waits for that packet to step and complete init. Without JSON sensor data it doesn’t send the MAVLink heartbeat over TCP 5760.

Socket state after a few sessions:

$ ss -ulnp | grep :9002
UNCONN  127.0.0.1:9002  users:(("ruby",pid=142033,fd=37))
UNCONN  127.0.0.1:9002  users:(("ruby",pid=79368,fd=57))

Two gz sim processes from different runs were bound to UDP 9002 (SO_REUSEADDR/SO_REUSEPORT). The kernel delivered packets non-deterministically — half went to a zombie process, lock-step never closed.

The orphans appeared because daemon.sh stop / a Ctrl-C on the tmux session didn’t reach the children. Linux reparents them to systemd --user, and they keep their bind.

Fix

scripts/shared/d2_sitl_cleanup.sh — --dry-run snapshot, --kill if needed. After every daemon.sh stop, always call the helper. The pre-flight port check in launch.sh catches an occupied MAVLINK_PORT (via TASK-033 ph2), but there’s no such check for UDP 9002 — SO_REUSEPORT silently allows the bind.

Verification 2026-05-08: killed 8 orphans via --kill, started a clean stack — JSON arrives (JSON received: timestamp/imu/position/quaternion/velocity), but arducopter still stalls after validate_structures and doesn’t send heartbeat. Orphans were a contributing factor, not the only cause.

Cause 2 — Cyrillic comments in indoor.parm (dev-log/05)

Enabled ENABLE_DEBUG = 1 in libraries/AP_Param/AP_Param.cpp:62, rebuilt SITL. In the output:

Ignored unknown param а in defaults file .../indoor.parm

(а is a lone Cyrillic ‘a’.)

What’s happening in the parser (AP_Param.cpp)

• Line 2272: if (line[0] == '#') return false; — comments are detected only at the start of a line.
• Line 2325: char line[100]; — 100-byte buffer.
• fgets(line, 99, file) reads ≤99 bytes per call. A comment longer than 99 bytes splits across multiple fgets calls — only the first one has # at position 0. The next chunk starts with text → parsed as an unknown parameter.
• One unknown param: done_all_default_params = false.
• Lines 1722–1727: load_object_from_eeprom() calls reload_defaults_file(false) every time a new var subtree registers AND done_all_default_params == false.
• → infinite reload loop, arducopter never reaches the GCS scheduler → no heartbeat.

UTF-8 makes Cyrillic 2× costly per character. Even a 50-character Russian comment easily overruns the 99-byte buffer. ASCII under ~80 characters fits safely.

Fix

simulation/config/ardupilot/indoor.parm — all comments in ASCII, wc -L = 78.

Bonus finding — ARMING_CHECK → ARMING_SKIPCHK (ArduPilot 4.7)

While debugging, I tried diagnostic-minimal.parm with ARMING_CHECK 0 — also unknown. Source:

• libraries/AP_Arming/AP_Arming.cpp:128 — // 2 was the CHECK paramter
• libraries/AP_Arming/AP_Arming.cpp:201 — AP_GROUPINFO("SKIPCHK", 13, ...)
• In init(), the migration: PARAM_CONVERSION - 4.7 CHECK -> SKIPCHK.

Our indoor.parm already used ARMING_SKIPCHK — wasn’t blocking. Noted for future reference.

Verification

With a clean indoor.parm the reload-loop disappeared, but the heartbeat still doesn’t arrive. Stall after validate_structures: Validating structures. This isn’t our indoor.parm and isn’t our world — same behavior with the canonical ardupilot_gazebo/worlds/iris_runway.sdf and without --add-param-file.

Cause 3 — --console flag → MAVProxy matplotlib hang (dev-log/06)

After two fixes it’s still stuck. Walked through bisection:

1. ArduPilot master regression — checked out the stable Copter-4.6.3 tag → same stall.
2. World missing <spherical_coordinates> — added → still stall (but kept the block, it’s needed for navsat/AHRS init).
3. indoor.parm content — diagnostic-minimal.parm (only ARMING_SKIPCHK 1 + BATT_MONITOR 0) → still stall.
4. iris_claudedrone model — swapped for canonical iris_with_gimbal → heartbeat in 4 s. False positive: stripped sg90 → works, stripped sensors_link → works, full model → no heartbeat. “Model broken” hypothesis didn’t reproduce until one variable changed.
5. launch.sh vs direct sim_vehicle.py — model-bisect tests used sim_vehicle.py --no-mavproxy, while launch.sh uses --console and lets sim_vehicle.py launch MAVProxy itself. Direct with --console reproduces the stall. Direct without --console — heartbeat in 3 s.

What --console does

Launches matplotlib via MAVProxy’s “console” UI module. In the output was an ignored warning:

/home/aleks/venv-ardupilot/lib/python3.12/site-packages/matplotlib/projections/__init__.py:63:
UserWarning: Unable to import Axes3D. This may be due to multiple
versions of Matplotlib being installed...

Not fatal, but MAVProxy’s “console” wedges on a downstream import after this path, never returns control to the heartbeat loop. arducopter is running fine (backtrace: AP_Scheduler::loop() → wait_clock → JSON::recv_fdm → select — normal lockstep wait); it’s just that its messages went through MAVLink (which MAVProxy was supposed to render), not to stdout.

Fix

simulation/help_scripts/launch.sh — remove --console from cmd_sitl(). MAVProxy works in plain terminal text mode, prints heartbeats inline.

End-to-end: ./help_scripts/launch.sh --sim --headless -d -log → heartbeat in ~6 s, mode STABILIZE, AP: ArduCopter V… Frame: QUAD/X. Stable on master (4.8.0-dev) and Copter-4.6.3.

Why the dev-log/04 hypothesis tree didn’t catch this immediately

dev-log/04’s candidate list: lock-step mismatch, EKF init, plugin schema vs data — all three misdirections. Autopilot was healthy. Real diagnosis became possible when:

1. I waited long enough (~30-40 s, not 25) to see whether heartbeat starts on canonical setups.
2. I compared two sim_vehicle.py invocations side-by-side, rather than trusting launch.sh output as ground truth.

Rules for future SITL bring-up debugging

1. First check — pgrep -af "gz sim|mavproxy|arducopter" and d2_sitl_cleanup.sh --dry-run. The orphan UDP 9002 race is a real failure class.
2. .parm files — ASCII only, wc -L ≤ 78. The AP_Param 100-byte buffer is a real constraint.
3. Compare direct sim_vehicle.py --no-mavproxy with launch.sh before blaming world/model.
4. No --console flag in production launch. Plain terminal mode is enough.
5. <spherical_coordinates> in world.sdf — pre-emptive hygiene for navsat/AHRS, add it right away.
6. If debug-needed — ENABLE_DEBUG = 1 in libraries/AP_Param/AP_Param.cpp:62, rebuild SITL. It’s a lazy hack — return it to 0 afterward.

Sources

• ~/drone_media/sim/TASK-021/21a_sitl.log — stall and recovery
• ~/drone_media/sim/TASK-021/21a_mavros.log — FCU: Connection problem → connected: true
• simulation/help_scripts/launch.sh (historical commit) — --console removed
• simulation/config/ardupilot/indoor.parm — comments rewritten in ASCII