California’s self-driving car reports are imperfect, but they’re better than nothing

On Wednesday, California’s Department of Motor Vehicles released the latest batch of reports from companies testing self-driving vehicles in the state. The reports offer the public its best glimpse into the slow and often-opaque process of testing autonomous vehicles on public roads. But because companies have a lot of leeway in how they report their data, the story it tells can be misleading.

By law, all companies that are actively testing self-driving cars on public roads in California are required to disclose the number of miles driven and the frequency in which human drivers were forced to take control of their driverless vehicles (also known as a “disengagement”).

The DMV defines disengagements as “deactivation of the autonomous mode when a failure of the autonomous technology is detected or when the safe operation of the vehicle requires that the autonomous vehicle test driver disengage the autonomous mode and take immediate manual control of the vehicle.”

This vague definition has allowed AV companies to avoid reporting certain events that, depending on how you look at it, could qualify as a disengagement. Last year, for example, a self-driving car owned by GM Cruise ran a red light in San Francisco after the safety driver took control to avoid blocking a crosswalk. But the company didn’t include the incident in its report because, according to Cruise’s interpretation, the human driver didn’t act out of safety concerns or a failure of the autonomous system.

Without strict definitions of when a safety driver should take over and more granular information from every company about where and when testing is happening, critics say there is no basis for comparison. A mile driven in Palo Alto isn’t the same as a mile driven in San Francisco. Similarly, if a company is focused on highway driving, it should inherently have far fewer disengagements. Even the time of day — rush hour versus early in the morning — would make a difference.

This broad interpretation of the rule has led some to dismiss the disengagement reports all together. “They are all utterly meaningless,” said Sam Abuelsamid, a senior analyst at Navigant Research.

Others see some value in the reports. “Absent any other metrics, they are an interesting portal into the significance of effort and general progress being made,” said Michael Ramsey, senior research director at Gartner.

If anything, it’s a good snapshot of the race between the two top performers: Alphabet’s Waymo and GM’s Cruise. Both posted significantly more miles driven in 2018 compared to the prior year and lower disengagement rates.

Photo by Vjeran Pavic / The Verge

Waymo drove 1.2 million miles in California in 2018, more than tripling the company’s 352,000 miles in 2017. To be sure, Waymo drove fewer miles in 2017 as the Google spinoff shifted the majority of its fleet to Arizona, in anticipation of the launch of its first commercial taxi service. That service went live (sort of) at the end of 2018, and in the meantime, Waymo has redoubled its efforts in its home state. The company’s disengagement rate dropped to 0.09 per 1,000 self-driven miles, or one per 11,017 miles.

Waymo says its lower disengagement rate indicates that its cars are getting better at handling “edge cases,” or situations that human drivers may encounter only once — or never — in their lifetimes. This will prove useful as Waymo grows the size of its fleet and expands into new territories, the company said in a blog post. (Waymo did not disclose the number of vehicles it has in its California fleet in its report to the DMV.)

“The way to keep learning and improving so this technology can scale is to give our self-driving vehicles as many robust and varied types of experiences, and uncover as many of those edge cases, as we possibly can,” Waymo says.

Image: Cruise

GM Cruise reported driving 447,621 miles in 2018, also more than tripling its total in 2017. The company reported 86 disengagements, or one every 5,205 miles. That translates to 0.19 disengagements per 1,000 miles. Cruise says it is operating 162 self-driving vehicles in California.

AV experts like to say it’s not the total number of miles driven autonomously, but the quality of those miles. Cruise’s vehicles operate primarily within San Francisco city limits and on roads that are arguably are more complex than Silicon Valley towns such as Mountain View, Palo Alto, and Cupertino. “Based on our experience, every minute of testing in San Francisco is about as valuable as an hour of testing in the suburbs,” Kyle Vogt, chief technology officer at Cruise, said in a Medium post in 2017.

Sneaking into third place is Apple, which reported 79,745 miles and 6,951 disengagements, or about 1.15 miles per disengagement. That’s a lot of interventions, and Apple is certainly playing catchup compared to Waymo and GM Cruise. The report shows that Apple had 62 self-driving cars on the road, but other data has suggested Apple is closer to the 70-car mark. In the past, Apple has criticized the reporting requirements, urging the DMV to “amend or clarify” its position on disengagement and testing without safety drivers.

It’s a tumultuous time for Apple’s secretive self-driving program, codenamed “Project Titan.” Last month, an Apple employee who is a Chinese citizen was accused by the FBI of attempting to steal trade secrets related to the company’s autonomous car project, marking the second time the government has charged an Apple employee for trying to steal self-driving secrets in the last seven months. Apple also recently laid off about 200 employees from the project.

Other startups making progress include Zoox (0.50 disengagements per 1,000 miles), Nuro (0.97 disengagements per 1,000 miles), and Pony.ai (0.98 disengagements per 1,000 miles). Tesla continued its tradition of reporting no miles driven in autonomous mode. (The electric car company has said it conducts “shadow-testing” of autonomous tech by collecting data from its Autopilot-equipped cars that are already on the road during normal operations.)

In total, 48 companies reported driving 2 million miles in autonomous mode on California roads and highways, the DMV said. But without a standard process or a way to verify the accuracy of the data, it’s not clear the reports are doing much to improve the public’s trust in the technology.

Polls show most people are still skeptical about self-driving cars, and aren’t necessarily clamoring to give up control. More transparency around testing would go a long way toward improving the public’s attitude toward robot cars. But until the government steps up and demands more from AV operators, these DMV reports are really all we have.