From Millions to Billions: Why Traditional Testing Can’t Keep Up with Autonomous Driving

Before an autonomous system can be released, it has to prove its safety to regulators, society, and the manufacturer.

Back when I was Chief Product Owner for Simulation, Software-in-the-Loop, and V&V toolchains on an SAE Level 4 project, I faced a tough question constantly:

How much real-world driving is enough to show that a self-driving system is at least as safe as a human?

The intuitive answer is usually millions of kilometers. But this intuition is misleading. When you look at actual road safety statistics, the reality is different. We aren't talking about millions. We are talking about billions of kilometers.

Why Billions? The Numbers Don’t Lie

Let’s look at the data:

• In many developed regions, fatality rates are only a few deaths per billion vehicle-kilometers.
• Injury accidents are more common, but still rare enough that testing purely by mileage struggles to provide statistically meaningful results.

If we want 95% confidence that an autonomous system is at least as safe as a human, the required exposure skyrockets. Estimates suggest tens to hundreds of billions of kilometers using traditional road tests, depending on the assumptions. Covering even tens of thousands of kilometers per feature is nowhere near enough, especially for a full Level 4 system that must operate across a broad Operational Design Domain (ODD).

Why Hitting the Road Isn’t Enough

The classic approach goes like this:

1. Develop the system.
2. Drive millions of kilometers.
3. Watch for accidents or near-misses.
4. Fix software.
5. Repeat.

Here’s why it falls apart:

• 1,000 vehicles × 50,000 km/year = 50 million km/year
• To reach 2 billion km → 40 years
• To reach 100 billion km → 2,000 years

And every software update can invalidate prior mileage as evidence, because the system you validated is no longer the system you are shipping. Traditional testing is basically only feasible for Level 2 systems, where a human driver is always available as fallback.

Why Current Workarounds Aren’t Enough

The industry has tried several shortcuts:

• Simulation: Test millions of kilometers virtually.
• Scenario databases: Focus on critical situations instead of raw mileage.
• Rare-event statistics: Estimate failures without waiting for accidents.
• Fleet learning: Use logged driving data to improve coverage.

All helpful—but none alone can fully guarantee safety. Simulations can’t perfectly replicate reality, scenario databases can miss unknown edge cases, and statistical models depend heavily on assumptions and data quality. Even massive real-world fleets struggle to cover long-tail, high-risk situations quickly enough to support credible safety claims.

The answer is combining these methods in a continuous, scenario-focused loop, where simulation, scenarios, rare-event methods, and fleet feedback all contribute to the Validation decision. Only then can we reach the scale and confidence that Level 4 demands.

So… How Many Kilometers Are Actually Enough?

Whatever method you use—simulation, scenario generation, statistical modeling—the numbers all point to the same order of magnitude:

• Billions of km effectively “driven”
• Millions of simulated scenarios
• Billions of agent interactions

Accelerated evaluation lets us compress tens of millions of equivalent kilometers into just thousands of targeted test kilometers—while still producing meaningful safety evidence.

Reality Check

Even doubling the fleet or increasing annual mileage won’t change the fact that decades—or centuries—would be needed to accumulate tens to hundreds of billions of kilometers. Pure road testing alone simply cannot serve as the primary safety argument for Level 4 vehicles.

What’s Coming in This Series

This is only the tip of the iceberg. In the upcoming weeks, I’ll go step-by-step through the full Scenario-Centric Continuous V&V Loop:

Each piece will include practical examples, simplified demos, and insights drawn from real experience.

Why It Matters

Moving to scenario-driven, continuous V&V is not just a technical choice—it’s essential. At Level 3 and 4, classical testing collapses as a safety argument, and safety evidence must be synthesized from simulation, logged data, scenario generation, and formal methods.

Virtual V&V at scale provides regulatory-grade safety arguments while keeping development timelines realistic. Understanding this foundation is key before diving into search-based testing, scenario generation, and digital toolchains.

Takeaways

• Road testing alone can’t provide Level 4 safety assurance.
• Modern fatality data show the massive exposure needed for statistically valid testing.
• Combining simulation, scenario databases, rare-event methods, and fleet learning is the only viable path.
• Multiple independent methods converge on billions of effective kilometers.
• This series will unpack the full Scenario-Centric Continuous V&V Loop, from first principles to applied tooling.

Autonomous vehicle validation is now a hybrid of statistics, simulation, and continuous feedback. Over the next weeks, I’ll show how one can actually build these loops in practice, and how each component contributes to the big picture.