What this is: A debugging story. For weeks our simulated drone kept flying in zig-zags instead of clean straight lines, and we finally chased the cause down to the root. This is the autopsy.
Why it’s here: Heading drift is one of those bugs that hides in plain sight — every individual turn looks “almost right,” and the error only shows up after you stack many of them. I want to leave a clear trail for the next person who sees a drone wandering off its grid.
Date: 2026-06-07 Ticket: Heading-drift RCA (simulation dev-log #27)
Glossary
- Heading drift — the slow, accumulating error in which direction the drone is facing. Imagine asking a friend to “turn 90 degrees” ten times in a row, but each turn they only manage about 73 degrees. No single turn looks wrong, yet after ten turns they’re facing a completely different wall than you expected. That gap, building up turn after turn, is heading drift.
- RCA (Root-Cause Analysis) — the discipline of not stopping at “the drone flies crooked” but asking why, then why that, until you reach the one thing that, if fixed, makes the symptom disappear. It’s the difference between mopping the floor and finding the leaking pipe.
- Yaw — rotation around the vertical axis: the drone spinning left or right while staying level, like a person turning on a swivel chair. (The other two are pitch — nose up/down — and roll — tilting side to side.)
- Open-loop rotation — turning by command-and-hope: “spin at this rate for this many seconds.” Nobody checks the result mid-turn. If the physics under-delivers, the error just stands.
- Closed-loop (absolute) setpoint — turning by aiming: “face exactly 90 degrees, and don’t report done until you’re actually there.” The controller corrects itself, so the final heading is what you asked for.
- The grid — our world is laid out so the drone’s “home” headings sit on a tidy lattice of fixed angles. Staying on the grid keeps flight paths square and predictable; sliding off it is where the trouble starts.
1. What we wanted
The drone was mapping a room and the tracks looked wrong. Instead of crisp straight sweeps along the walls, we kept getting zig-zags and odd diagonal approaches — the drone shuffling up to a wall at a slant, doing little “no-travel” wiggles, then moving on.
The open question was uncomfortable: is this the navigation model misbehaving, the position estimate being noisy, or something more mundane? Before touching any model logic, I wanted to know which of those it actually was. A guess here would have sent us refactoring the wrong subsystem for days.
The working hypothesis (shaped together with Aleks): the world lives on a heading grid — a set of evenly spaced angles 15 degrees apart. Each reset drops the drone at a random heading, and from there it’s supposed to rotate in exact 15-degree steps to stay aligned. But if the in-sim rotation under-rotates — delivering, say, ~12 degrees when we command 15 — the drone slides off the grid a little on every turn and never climbs back on. The prediction was specific and testable: the drift should be systematic (always the same direction, not random jitter) and should accumulate with each rotation.
That specificity is what made it falsifiable. Random noise would have wandered both ways and averaged out; a real defect would march steadily one way.
2. What we did
Run A — observe the drift (snap OFF)
First, just measure. I ran the standard empty-room scenario and logged every rotation.
- Per-rotation calibration: across 19 rotations, every one commanded as a 15-degree turn, 17 of 19 under-rotated. The mean error was about +2.84 degrees of shortfall per rotation, with turns landing at roughly 82% of the commanded angle. That’s not noise — noise doesn’t favor one direction 17 times out of 19. It’s a systematic bias.
- The drift curve over time was the clincher. The cumulative heading error climbed monotonically: 0 → +0.86 → +2.24 → +3.49 → +4.37 → +6.73, then wrapped around to −7.48. By the tail, the accumulated misalignment was about 7.46 degrees — half a grid cell. The drone was effectively stuck between two grid lines, aligned with neither.
- Looking at the approach headings for the wall-mapping action, the drone was starting its approaches at −14°, +17°, −27°, and even −38.6° off the room axes. Every one of those slanted starts produced a “no-travel” wiggle. The “approaching the wall at ~30 degrees” picture was confirmed exactly.
- Run A bottom line: the room reached 96.2% mapped, but it took 382 steps and 238 stall recoveries to get there. Slow, messy, lots of stuck moments.
Here’s roughly what the Run A track looked like — diagonal, jagged approaches:
Run B — the control experiment (snap ON)
Then the test that settles it. I added one switch: before each wall-mapping action, snap the heading to the nearest 90 degrees using a closed-loop absolute setpoint — aim at the exact angle and wait until the drone has actually arrived, rather than spinning open-loop for a fixed duration. Everything else was held identical. This is the heart of an RCA: change exactly one thing, and see if the symptom dies.
Here’s the Run B track — clean lines, square approaches:
And, as a small first for us, an animated map-plus-route showing the run unfold:
3. Results
The two runs side by side:
| metric | A (snap off) | B (snap on) |
|---|---|---|
| reached 95% mapped at | step 382 | step 125 (~3x faster) |
| final mapped | 0.962 | 0.978 |
| heading error (mean / tail) | 5.20° / 7.46° | 0.84° / 0.85° |
| stall recoveries | 238 | 19 |
| track shape | zig-zag, slanted approaches | straight lines on the axes, clean 90° turns |
The numbers aren’t subtle. Snapping the heading back onto the grid hit the mapping target about three times faster, finished more complete, and cut stall recoveries from 238 down to 19. The accumulated heading error collapsed from over 7 degrees to under one. And the tracks went from a jagged scribble to clean square sweeps.
Root cause closed: the zig-zag is heading drift from open-loop yaw-rate rotations. It was never the navigation model, and never the position estimate. The drone was simply flying at an ever-growing angle to the grid, and that angle showed up on every track and every video as a zig-zag. One control experiment, one clean answer.
4. Consequences and caveats
A few honest notes so nobody over-reads the win:
- A clean fix exists (a candidate design we discussed): make the relevant rotation actions use a closed-loop absolute yaw setpoint onto the grid, instead of open-loop rate-times-duration. Whether and when to ship that is a separate decision.
- This isn’t the only factor. Run B still showed a handful of no-travel moments and feasibility masks. The margin-band behavior remains a second, independent issue — fixing heading drift didn’t make it vanish, and I don’t want to pretend it did.
- Speed was deliberately not touched. To keep the A/B comparison honest, I changed nothing about flight speed between the two runs. The speed cap stayed where it was. Raising it belongs in its own separate change, after the rotation decision is made — mixing it in here would have muddied which improvement came from what.
5. Sources
| What | Where |
|---|---|
| Track PNGs | tracks/track_driftA_empty6x6_20260607.png, track_driftB_...png |
| Animated map + route (our first) | tracks/map_driftB_empty6x6_20260607.gif |
| CSV logs (odometry + yaw / actions / metrics) | tracks/track_drift_{A,B}_*.csv, track_drift_B_occ.npz |
| Flight videos | sim_20260607_182846.mp4 (A), sim_20260607_184232.mp4 (B) |
| Runtime logs | _runtime_logs/sim-*-drift{A,B}.log |
| Distance-sensor fix verified | one publisher / one subscriber on the mavros range topic |
These all live in the simulation repo. The flight videos in particular are worth watching back to back — the difference between the two tracks is obvious even at a glance.
6. What’s next
The immediate follow-up is to decide on the closed-loop absolute-yaw rotation fix and land it as its own change, then revisit flight speed separately. The margin-band / no-travel behavior stays on the list as the next thing to chase down — now that heading drift is off the table, it should be easier to isolate.
For the toolchain record: this work sits in our standard Gazebo + SITL + ROS2 (Jazzy) setup, with mavros bridging to the ArduPilot flight stack and the EKF handling state estimation. None of those changed here — the lesson is purely about how we commanded the rotations.