You train a model overnight. In the morning it’s perfect: every mission completed, zero crashes, a flawless flight through every test map. You want to ship it immediately. And that is exactly the moment to stop and ask one uncomfortable question: was that real, or did I just get lucky?
In reinforcement learning (RL), a single great result is not proof. It might be skill. It might be chance. Telling the two apart is one of the most important — and most skipped — steps in building a system you can actually trust. This article explains how we do it, using a real case from our drone work: a model line we call scan-v1.
The core problem: one run proves nothing
A training run does not start from a blank, fixed state. It starts from randomness — where the policy’s parameters begin, the order in which experiences arrive, the small noisy choices made along the way. RL is built on exploration, and exploration is random by design.
That randomness has a knob, and the knob is called a random seed. A seed is a single number that fixes all the “dice rolls” inside one training run. Use the same seed and you get the same trajectory every time. Change the seed and the whole run unfolds differently — different starting point, different path, sometimes a different final result.
Here’s the useful analogy: same recipe, different dice. The recipe is everything you control — the algorithm (we use PPO, Proximal Policy Optimization), the data, the settings, the starting foundation model. The seed is the dice you roll on top of it. One training run is one roll of the dice. And with one roll, you genuinely cannot tell whether you have a good recipe or a good roll.
A perfect single run proves only that the recipe can produce a perfect model. It says nothing about whether it reliably does. Sometimes the dice just fall your way.
The validation method: rerun the recipe, change only the dice
The fix is simple to state and disciplined to do: rerun the identical recipe under several different seeds, then test every resulting model across every scene.
The word “identical” is doing real work here. You change nothing except the seed — same data, same settings, same foundation to build on. That isolation is the whole point. If the outcomes differ, the difference can only come from the randomness, because that’s the only thing you changed. If the outcomes hold up everywhere, the result is a property of the recipe, not an accident of one lucky roll.
The mindset is the same as checking that a recipe works on three different stoves, not just your own kitchen. If the dish comes out right on all of them, you’ve learned something about the recipe. If it only works on yours, you’ve learned something about your stove.
A worked example: scan-v1 → production
In an earlier run, one scan-v1 model flew through all seven of our training scenes perfectly: 100% of tasks completed, zero collisions. Beautiful — but it was one training run. One roll of the dice. Before promoting it to be the production model — the one we actually ship — we had to know whether that perfection would come back.
So we re-ran the exact same recipe from scratch with three more seeds. That gave four models in total. Then every model was put through every scene: four models across seven scenes, twenty-eight separate exams.
The outcome: all four models, on all seven scenes, completed 100% of tasks with zero collisions. Every one of the twenty-eight exams was a clean pass. The success reproduced. It wasn’t luck — and scan-v1 was promoted to the production model.
One detail is worth keeping, because it shows why you sanity-check even good news. The four models flew their routes almost identically. At first glance that’s alarming — maybe the seeds never took effect and we’d accidentally trained the same model four times, which would make the whole test meaningless. So we checked the file fingerprints of the model weights: they were genuinely different. The seeds had done their job. The models simply converged on the same near-unique flight path to the targets while differing in how they swept their sensors along the way. Different models, same good answer. That is exactly what reproducible success looks like.
For the full run-by-run account, see dev-log 53.
A checklist for trustworthy RL claims
When someone shows you an RL result — or before you make one yourself — run it through this:
- Use at least three seeds. One run is a roll of the dice; a small handful starts to reveal the recipe. More is better, but three is the floor for taking a claim seriously.
- Test on every scene, not the easy ones. A result that holds across all environments is a property of the model. A result cherry-picked from the scene where it happened to shine is marketing.
- Change only the seed. Keep data, settings, and the starting foundation identical, so any difference can only come from randomness — the thing you’re trying to measure.
- Report the failures too. A claim that hides the runs that didn’t work isn’t a validation, it’s a highlight reel. Trust comes from showing the whole picture.
- Sanity-check surprises, even good ones. Results that look “too clean” deserve a second look — confirm the experiment actually did what you think it did.
The takeaway
The distance between “one promising result” and “a recipe we trust” is exactly the work of multi-seed validation. It’s the difference between a lab curiosity and something you can build the next sprint on. A single flawless run is a reason to get curious, not a reason to ship. The honest answer to “is it real or just luck?” only arrives after you’ve rolled the dice more than once — and watched the result hold every time.