Kodiak Robotics recently made another first, not from a safety case or new route but from its virtual driver being scored as if it were a human one. According to Nauto, a fleet safety technology provider, Kodiak’s virtual driver, called the Kodiak Driver, passed with flying colors.
The safety scoring system is called VERA and stands for Visually Enhanced Risk Assessment. It’s Nauto’s proprietary safety benchmark that looks at human driving behaviors to determine a driver’s safety over a period of time. The Kodiak Driver scored a 98 out of 100 possible points among more than 1,000 commercial fleets in Nauto’s network.
For human drivers, there was a silver lining: Kodiak’s virtual driver ended up tying with a human fleet that also scored a 98. For an average driver, according to Nauto, their VERA score is 78 for a fleet with Nauto’s safety tech, and fleets without it scored an average of 63. But a key question is what the scores mean, and how this bodes for other autonomous trucking companies that, at the moment, are only using third-party safety cases and validations.
To answer these questions, FreightWaves interviewed Dr. Stefan Heck, CEO and founder of Nauto, along with Don Burnette, founder and CEO of Kodiak Robotics, to discover what this means and what the future looks like for measuring the performance of these new robotic drivers.
What’s VERA and what makes it special
VERA was born out of necessity nearly 10 years ago. “Our goal really was very simple: to give the fleet a way to identify who’s a great driver, who’s a decent driver, and who’s at risk. That’s what we built it for originally,” said Dr. Heck.
Heck notes that it’s a score based on the risk of a particular trip. “We called it VERA because we wanted to highlight that visual data is being used in the scoring, to contrast it with traditional telematics scores, which are really just doing things like counting hard braking or counting how often you speed,” said Heck.
Back in the day, the traditional way to score your driver looked at vehicle movements. VERA added a few extra steps including such things as if a driver ran a stop sign, did they hit a person, how close were they tailgating, etc. These things over 10 years ago were mostly unheard of. Nowadays they’re table stakes, but their appearance in the telematics space is a recent phenomenon, borne of the rapid innovations of technology being cheaper, meaning hardware is cheaper, and now the way to transmit the data to figure it out is also cheaper.
For Kodiak, which uses Nauto’s technology, the opportunity to have its virtual driver be graded like a human one was a unique opportunity.
“Now, this particular effort came because Kodiak approached us—and they probably told the story. They said, ‘Hey, could we use your scoring system to evaluate our autonomous driver?’ And we were like, ‘Oh, that’s a really interesting idea. We’d love to do that.’ For the human driver, you’re just observing their behaviors. Even though you can’t look inside the head of the autonomous driver either, we can see what the vehicle is doing. We can score the same way we score the driving behavior,” said Heck.
This is a step up compared to the current autonomous truck playbook of having a safety case validated by a third party showing you know what to do and your truck performs as expected. Heck notes there are some downsides to only safety cases, mainly that it doesn’t capture the trucking equivalent of a “black swan event.”
“The safety cases, from my point of view, are incomplete for one very simple reason: If you look at collisions generally, they’re pretty rare. Serious collisions are 100 million miles apart; fatalities are even rarer. So unless you really have billions of miles of data, you can’t really say for certain that the new AV driver was really good versus ‘got lucky,’” Heck said.
Another challenge with AV providers is they cherry-pick what they show. Heck notes, “They publish some of their data, but they don’t show you all the time—does the human override the autonomous driver and take control back? That should really be part of it.”
One example of showing your work, Heck gave, was Waymo. “I think Waymo does a good job. Waymo publishes more data and shows how many times the supervisor or the remote driver or the human safety driver took over. So you start to get a sense, and they’re definitely getting really good as well.”
This is where the VERA score comes in handy. As AVs don’t have as many miles of real-world driving time, it’s hard to know for certain how they would react to very rare instances that were not considered in their robotic programming.
For human drivers it’s not all doom and gloom
For drivers, concerned citizens or even neo-Luddites, the Kodiak Driver’s score is not all doom and gloom. After all, it was only two points short of a perfect score, and it may be nearly impossible to get a perfect score, partially because a robotic truck at the moment still has to deal with other human drivers of various skill levels.
“I’m not surprised at all that the Kodiak driver on that dimension didn’t get a perfect score, because you’re still driving in traffic where you have to deal with other things that you’re responding to. I think there are two ways to read the result here. One is clearly—and I think this is the great news for Kodiak—they’ve managed to build a driver that is better than many human beings. The other good news story here is you can definitely get human drivers to drive at that same level of capability,” said Heck.
Similarly, Don Burnette, Kodiak’s founder and CEO, noted that when in a chaotic human-driven world, sometimes the best strategy is to focus on safety and be prepared for the unpredictability of human traffic.
“The truth of the matter is, we believe we have developed a very conservative driver, and our truck mainly sits in the right-hand lane, minding its own business. The trucks do ultimately have to adapt to the traffic around them, and sometimes you have to deal with harsh cut-ins, merging scenarios, and other aggressive driving maneuvers from other vehicles that require uncharacteristic movements of our trucks. We believe we are at the top of what is considered best practices for driving and at the end of the day that’s what we really focus on,” said Burnette.
Heck adds that with these results, don’t expect there to be a compelling reason anytime soon to get rid of human drivers, one key reason being that for many companies, the technician who does the work also has to do the driving. “So it is not an argument for ‘Hey, get rid of all the humans and only have Kodiak drivers,’ but you can certainly trust the Kodiak driver to do a good job. Anybody who has human drivers—and we work with a lot of fleets where the driver isn’t just a driver… where the person’s in there to do some other job… the person isn’t going to go away.”
“For those companies and those fleets, it’s really great news that you can get the human driver to be as good as well. You can kind of approach it either way: AV gets you great safety benefits; having AI make your human drivers better also gets you great safety benefits. It’s fantastic,” said Heck.
“At the end of the day, we think what the motoring public wants to know is: are autonomous vehicles safe, and are they well-behaved drivers, the way that you would hope human drivers would be. What is exciting about this study is it shows autonomous vehicles do in fact behave like the very top half of the best human drivers, which is what our industry has historically claimed. And it’s nice to have third-party validation that this is true,” said Burnette.
