Tesla’s Autopilot safety stat is seriously flawed

It turns out that a heavily cited safety stat that Tesla received from the National Highway Traffic Safety Administration (NHTSA) back in 2017 isn’t so accurate.

Following a deadly Tesla crash involving Autopilot, Tesla’s semi-autonomous advanced driver assistance system, a 2017 federal report found that Autopilot led to a 40 percent crash reduction. But last week a small research group finally obtained most of the data NHTSA used to reach that number and found the underlying data was severely flawed.

The discovery confirms what the group, Quality Control Systems Corp., and others like this report from Wired last May. People had been skeptical for a long time. Even NHTSA said last year the numbers weren’t as solid as they had originally indicated. The QCS researchers wrote in their report, “Remarkably, NHTSA’s announcement was not accompanied by any of the data underlying this astonishing claim.” So back in 2017 they requested the data and successfully fought for access. 

Autopilot software and hardware has changed since the report came out, so this doesn’t have much bearing on how Tesla cars function now. Tesla continues to earn high marks in crash safety tests. It’s also been years since that data was collected. Tesla’s since hit 1 billion miles on Autopilot. The number of Tesla vehicles on the road has ballooned in the past few years. 

But now, measuring claims of increased safety on Autopilot and using features like Autosteer that keep the car in the lane isn’t as clear-cut. Now that QCS has shown that NHTSA made crash rate calculations using a Tesla data set with missing data and other flaws, Autosteer can’t be as confidently earmarked as a must-have safety feature.

Ars Technica delves further into the math and problems calculating the crash rate if you want to understand the denominators and mileage calculations. The publication also points out other factors affecting safety, like Tesla driver age and socioeconomic status. 

Tesla does voluntarily put out its own safety reports — though those are pretty brief and not-so-surprisingly missing any data.