What’s A Virtual Mile Worth To Self-Driving Safety?

Image for post
Image for post

Overlooked in Waymo’s recent claim that the eight millionth mile had been reached by their road-roaming test fleet was a less tangible milestone: the company has now cranked five billion virtual miles through its simulator. In a sector starved for comparison metrics, conventional logic would equate this digital feat to roughly 5,000 lifetimes of human driving.

For another perspective, it’s well over a million times the experience demanded of teenagers before they receive a driver’s license and a two-ton projectile. A reasonable comparison? Difficult to say. But trivial? Not in the least.

Simulators have long been an advantage for racing drivers, a supplement for commercial truckers, and a requirement for airline pilots. It stands to reason, then, that virtual miles driven by AI should count for something.

For starters, the cyberspace built by self-driving firms like Waymo and NVIDIA offers an opportunity to test and retest ad nauseam, in any manner of environment, with all manner of distraction and obstruction; some of these scenarios could take human drivers a lifetime to encounter just once.

Additionally, every terrestrial tribulation encountered by the cars’ sensors can be plugged into these sims, replayed to taste, and eventually disseminated across an entire driving fleet once successful behaviors are learned. That last part is worth repeating, as it is constantly underestimated: the AV you’re sitting next to at a red light might be getting better at driving right now, because its brother recently learned something in the sim and distributed that knowledge to the entire fleet.

Image for post
Image for post

Acknowledging virtual mileage as a near-1:1 replacement for on-road mileage makes sense in that light, and the repercussions of such a decision would be a remarkable reversal of conventional back-of-the-napkin math evaluating the safety of self-driving cars. For instance, RAND Corporation serendipitously projected that autonomous vehicles would need to travel five billion miles in order to demonstrate a 20% improvement in fatality rates at 95% confidence. With on-road driving, this would take centuries of effort (unless of course, the effect size turns out to be larger). With virtual driving, Waymo already did it.

And yet, the purpose of any training environment is to produce successful outcomes in reality. It could be argued that any credit for simulated miles is a sort of double-counting, as the AI’s behavior on public roads ought to be a reflection of its virtual mastery. So perhaps, virtual miles are worth nothing. Of course, placing the burden of proof solely on-road gets back to the point raised by RAND: there is no performance data human drivers produce with enough frequency to yield a practical benchmark for autonomous vehicles. Or it should be said, there is definitely performance data available, but because we don’t observe human drivers until a crash is reported, that data isn’t accessible. Ergo, machines are held to human standards, despite humans refusing to be standardized.

It’s a conundrum muddied even further by the reminder, again, that the U.S. feels completely comfortable putting car keys in the hands of people whose training models come nowhere near a statistically rigorous level. If anything, the relative danger of young drivers is well-established and proves they have no business getting behind the wheel until they’ve trained further.

Perhaps humans see roads as the great equalizer; a testbed so ubiquitous that the benefit of direct observation more than makes up for the temporal limitations and threat of catastrophe. In contrast, a virtual world designed by the very people trying to succeed in it is a black box on an honor system, rendered useless without a window in which society can peek a watchful eye.

Should that be the public’s sentiment, it would only prove the case for public testing to be self-evident: if simulated miles are unacceptable as training currency, then the only way to train driverless cars is to put them on the road, and accept their statistically insignificant mistakes as if they were America’s very own sons and daughters.

This article first appeared on Forbes.com.

Future of work, future of mobility, future of ice cream.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store