Toyota Story Analysis

I’m not against Toyota, really. I’m against manufacturers who try to weasel out of their responsibilities after they put people at risk with poor design. I do not know if Toyota is guilty of that, here.

I also don’t know what the real story is with this Sikes fellow who was in the car that he claims went out of control. Maybe he’s a gold digger out to exploit Toyota’s bad fortune.

Finally, I don’t know much about the design of a Prius, except that my understanding is that there are no physical cable linkages. It’s software driven. When you press the brakes you are essentially double clicking on the “brake” icon with your foot mouse, hoping that the operating system agrees to apply the real brakes. A Prius is basically a video game console connected to a car. You drive the console, not the car.

What I’m trying to do is use this situation to teach testing. So, I want to break down a news story I found on cnn.com and show you how a tester would think about it:

Toyota takes aim at California runaway Prius story
by Peter Valdes-Dapena, senior writerMarch 15, 2010: 8:24 PM ET

NEW YORK (CNNMoney.com) — Toyota challenged a California driver’s story of an out-of-control Prius at a press conference Monday afternoon.

Toyota held a press conference about this. That is interesting. Of course that means Toyota’s marketing people are actively involved in this investigation, pacing and fretting for good news to send out. This does not create a conducive atmosphere for an investigation.

Public relations people want to rush out with good news, but keep bad news longer to study it and make very sure it’s really correct. This creates a sort of chromatic distortion of the truth in the near term. We must beware of that. You might say that it’s likely to be “green shifted” truth.

Company executives detailed preliminary findings of a joint investigation conducted by Toyota and the National Highway Traffic Safety Administration into the incident.

I like hearing that the NHTSA is involved in the investigation. Are they leading it, I wonder? You know there are several ongoing investigations into Toyota vehicles by the NHTSA (see their website) including one involving the momentary loss of braking on uneven surfaces (which my father has experienced several times on the dirt road going to his house.)

I feel there’s more credibility if the NHTSA is involved, but the press conference seems to have been a Toyota thing, not an NHTSA thing.

Prius owner Jim Sikes made national headlines last week with claims that his car’s accelerator got stuck as he sped up to pass a car while traveling on California’s I-8 highway outside of San Diego, and that he was unable to stop the car.

“As I was going, I was trying the brakes … and it just kept speeding up,” he said.

Reports from non-technical, non-expert users always must be taken with skepticism, even leaving aside the possibility that the guy is just telling lies. Perhaps he suffers from Munchhausen syndrome. You never know.

But let’s say he’s not lying, just for a moment. The phrase “it just kept speeding up” suggests that the brakes were completely inoperative. I wouldn’t use that phrase, as a driver, if the brakes had engaged and were fighting the motor.

Someone needs to sit the guy down (I assume they’ve done this) and walk through the whole incident moment by moment. Several times. Get him to clarify this.

Inconsistencies alone should not worry us too much. It happened quickly, it was a traumatic event, and his personal account may not be reliable just for that reason. But we still need to wring every bit of information we can from his memory.

Sikes story is at odds with the findings of the investigation, according to Toyota and to a draft congressional memo obtained by CNN.

“While a final report is not yet complete, there are strong indications that the driver’s account of the event is inconsistent with the findings of the preliminary analysis,” Toyota said in a prepared statement.

Sikes said he called 9-1-1 for help as he was traveling in excess of 90 mph on a winding, hilly portion of the interstate. He said dispatchers tried to talk him through ways to stop the car, but nothing helped.

I’d like to hear that 911 call in its entirety.

Eventually, a California Highway Patrol officer was able to catch up to Sikes and used the patrol car’s public address system to instruct Sikes to apply the brakes and the emergency brake at the same time. That tactic worked, and he was able to stop the car.

I’d like to know if Sikes had tried the emergency brake alone, before this. Had he tried both at once before this.

I can understand if he did not try both at once, because a normal driver would think if they don’t work individually, why would they suddenly work together? But  in a drive-by-wire system, everything is mediated by software, and software can get into strange states. It’s technically plausible that only with both brake controls activated the software could be bounced out of whatever strange trance it got into.

However, because driving a hybrid car like the Prius with both the gas pedal and the brakes simultaneously depressed would cause serious damage to the car’s electric motor and, possibly other systems, Toyota says the Prius is designed to prevent that from happening.

Of course it’s designed prevent that. But here’s a testing lesson: designing for prevention is not the same as preventing, because your design may have a bug in it.

All Toyota can say is that it was their intent to design the system to prevent that, and to the best of their knowledge that is how the system works… except in this case it didn’t work– unless the guy is simply lying or insane.

If the brake is pressed at the same time as the gas pedal, power to the engine will be reduced just as if the gas pedal had been released, the automaker said.

Unless, of course, there’s a malfunction of the system, which is exactly the issue under consideration.

During driving tests on Sikes’ Prius and on an identical 2008 Prius, the system operated as expected, according the report, preventing the car from pushing forward while braking.

“The system operating as expected” is not actually possible to determine, because they can’t see inside all the software and hardware to detect that every bit and electron is in the right place.

What they can say is that they detected no problems. Problems may be there, they just did not detect them.

If the visible, detectable problems we want to see are triggered by a transient event, such as a specific combination of foot presses, or perhaps there’s a two microsecond window of opportunity for two software events to happen simultaneously (such as have dogged the Mars rover missions ), then of course driving it around a parking lot is probably not going to reproduce the problem.

It is also possible that part of the problem involves a piece of physical equipment that was lodged or worn in a certain way at a specific temperature, and that condition no longer exists on the car in question.

When we try to reproduce problems, we often have to guess at the causes, and we may guess wrong.

If I were Toyota, I would treat this like an epidemiology problem. You interview people and make a list of absolutely everything that was going on. Did they have a cell phone? What kind? Where was it in the car? Where they using the cup holder? What drink was in the cup? Hot or cold? Was the air conditioner on or off? What was the setting?

Then you put all the data into a database and mine it for patterns.

Investigators are extremely meticulous when taking apart a car in a case like this, said Ed Higgins, a Michigan personal injury attorney who has been involved in automobile defect cases. They are aware their work will be subject to intense scrutiny, so they measure and document everything, he said.

That kind of care takes a lot of time. But it hasn’t been very long since the incident occurred. Have they also taken the software apart? Have they comprehensively reviewed the code? I seriously doubt that.

“I would think that any mechanical defect that would have allowed something to happen that otherwise could not have happened would have stood out like a sore thumb,” he said.

Unless it’s a transient interaction between a mechanical defect and an invisible state within the software.

The car also did not show damage consistent with the engine having been running at full throttle while the brakes were on, according to the report.

That suggests the brakes weren’t on, but not that Sikes wasn’t pushing on the brakes.

“Toyota engineers believe that it would be extremely difficult for the Prius to be driven at a continuous high speed with more than light brake-pedal pressure, and that the assertion that the vehicle could not be stopped with the brakes is fundamentally inconsistent with basic vehicle design and the investigation observations,” Toyota said in a statement.

Again, this all assumes normal circumstances and no transient failures. For the purposes of investigation, that belief is irrelevant.

It is already fundamentally inconsistent with the design of the product that ANY failure could occur. We’ve crossed that bridge, guys.

Remember, when flight 427 crashed, Boeing maintained for years that their rudder mechanism could not possibly have failed– until a new form of failure was discovered (“thermal shock”) and the specific failure reproduced in that very rudder assembly.

The car’s front brakes showed significant wear and overheating, Toyota said. That kind of wear and heat would be consistent with the brakes being lightly applied over a long period of time, executives said at the press event.

Data from on-board computers indicated that Sikes had applied the brakes, to some degree, at least 250 times during the 23 mile incident, Toyota executives said, and that the brakes worked normally each time.

Ooh, I love log files. I wonder what other patterns they can mine from that log file?

If the computers indicate that Sikes had applied the brakes, that shows they were getting some kind of signal from the brake mechanism, but not necessarily the correct signal. Therefore saying “the brakes worked normally each time” is completely unwarranted. Part of the system may have been working normally while another part was going haywire. There’s not way to tell after the fact, because “working normally” is not a detectable condition that gets logged in computer.

Every time you experience a problem in your software, your software, on some level, thinks it is doing the right thing. Software doesn’t “know” it’s misbehaving. It just does what it is told.

Edmunds has independently tested Prius cars similar to Sikes’ and confirmed that the engine would stay engaged if the brakes were only pressed lightly, but not hard enough to actually stop or slow the car, said Dan Edmunds, head of auto testing for the automotive Web site Edmunds.com.

He says “would”, but he should say “would, assuming that there is nothing wrong with the car that would cause it not to”

“If you’re just riding the brakes, it will ride the brakes,” he said.

“These findings certainly raise new questions surrounding the veracity of the sequence of events that has been reported by Mr. Sikes,” said Kurt Bardella, spokesman for Rep. Darrell Issa, R-Calif., and ranking member of the committee.

Sikes’ attorney, John Gomez, denied that the report proves his client was wrong about what happened to his car.

“The notion that they weren’t able to replicate it in this particular case tells us nothing,” he said. “They haven’t been able to replicate a single one of these.”

That’s right. Also, learn the phrase “transient failure mode” and press that point. There are plenty of examples in space missions and airliners of such failures.

Sikes has no plans to sue Toyota, Gomez said.

Gomez is also representing the family of Mark Saylor, a California Highway Patrolman who was killed, along with members of his family in a Lexus sedan that accelerated out of control. A preliminary investigation has found that the accelerator pedal in that car probably became trapped on an all-weather floor mat that had been incorrectly installed in the vehicle.

Toyota has issued a recall for several models, including Sikes’ Prius, to address possible floor mat entrapment. Sikes’ floor mat was not interfering with the accelerator, investigators found, and there were no signs the pedal had become stuck in any way, according to the report.

The investigators findings “suggest that there should be further examination of Mr. Sikes account of the events of March 8,” Toyota said in its statement.

Toyota spokesman Mike Michels also took issue with media coverage of the Sikes incident. Journalists sensationalized an admittedly dramatic event, he said, but the public would have been better served had reporters waited for all the facts.

“We need to let investigations take their course,” he said.

Yes indeed. And this investigation has not done that. Be careful what you wish for, Mr. Michels.