Toyota Story Analysis

I’m not against Toyota, really. I’m against manufacturers who try to weasel out of their responsibilities after they put people at risk with poor design. I do not know if Toyota is guilty of that, here.

I also don’t know what the real story is with this Sikes fellow who was in the car that he claims went out of control. Maybe he’s a gold digger out to exploit Toyota’s bad fortune.

Finally, I don’t know much about the design of a Prius, except that my understanding is that there are no physical cable linkages. It’s software driven. When you press the brakes you are essentially double clicking on the “brake” icon with your foot mouse, hoping that the operating system agrees to apply the real brakes. A Prius is basically a video game console connected to a car. You drive the console, not the car.

What I’m trying to do is use this situation to teach testing. So, I want to break down a news story I found on cnn.com and show you how a tester would think about it:

Toyota takes aim at California runaway Prius story
by Peter Valdes-Dapena, senior writerMarch 15, 2010: 8:24 PM ET

NEW YORK (CNNMoney.com) — Toyota challenged a California driver’s story of an out-of-control Prius at a press conference Monday afternoon.

Toyota held a press conference about this. That is interesting. Of course that means Toyota’s marketing people are actively involved in this investigation, pacing and fretting for good news to send out. This does not create a conducive atmosphere for an investigation.

Public relations people want to rush out with good news, but keep bad news longer to study it and make very sure it’s really correct. This creates a sort of chromatic distortion of the truth in the near term. We must beware of that. You might say that it’s likely to be “green shifted” truth.

Company executives detailed preliminary findings of a joint investigation conducted by Toyota and the National Highway Traffic Safety Administration into the incident.

I like hearing that the NHTSA is involved in the investigation. Are they leading it, I wonder? You know there are several ongoing investigations into Toyota vehicles by the NHTSA (see their website) including one involving the momentary loss of braking on uneven surfaces (which my father has experienced several times on the dirt road going to his house.)

I feel there’s more credibility if the NHTSA is involved, but the press conference seems to have been a Toyota thing, not an NHTSA thing.

Prius owner Jim Sikes made national headlines last week with claims that his car’s accelerator got stuck as he sped up to pass a car while traveling on California’s I-8 highway outside of San Diego, and that he was unable to stop the car.

“As I was going, I was trying the brakes … and it just kept speeding up,” he said.

Reports from non-technical, non-expert users always must be taken with skepticism, even leaving aside the possibility that the guy is just telling lies. Perhaps he suffers from Munchhausen syndrome. You never know.

But let’s say he’s not lying, just for a moment. The phrase “it just kept speeding up” suggests that the brakes were completely inoperative. I wouldn’t use that phrase, as a driver, if the brakes had engaged and were fighting the motor.

Someone needs to sit the guy down (I assume they’ve done this) and walk through the whole incident moment by moment. Several times. Get him to clarify this.

Inconsistencies alone should not worry us too much. It happened quickly, it was a traumatic event, and his personal account may not be reliable just for that reason. But we still need to wring every bit of information we can from his memory.

Sikes story is at odds with the findings of the investigation, according to Toyota and to a draft congressional memo obtained by CNN.

“While a final report is not yet complete, there are strong indications that the driver’s account of the event is inconsistent with the findings of the preliminary analysis,” Toyota said in a prepared statement.

Sikes said he called 9-1-1 for help as he was traveling in excess of 90 mph on a winding, hilly portion of the interstate. He said dispatchers tried to talk him through ways to stop the car, but nothing helped.

I’d like to hear that 911 call in its entirety.

Eventually, a California Highway Patrol officer was able to catch up to Sikes and used the patrol car’s public address system to instruct Sikes to apply the brakes and the emergency brake at the same time. That tactic worked, and he was able to stop the car.

I’d like to know if Sikes had tried the emergency brake alone, before this. Had he tried both at once before this.

I can understand if he did not try both at once, because a normal driver would think if they don’t work individually, why would they suddenly work together? But  in a drive-by-wire system, everything is mediated by software, and software can get into strange states. It’s technically plausible that only with both brake controls activated the software could be bounced out of whatever strange trance it got into.

However, because driving a hybrid car like the Prius with both the gas pedal and the brakes simultaneously depressed would cause serious damage to the car’s electric motor and, possibly other systems, Toyota says the Prius is designed to prevent that from happening.

Of course it’s designed prevent that. But here’s a testing lesson: designing for prevention is not the same as preventing, because your design may have a bug in it.

All Toyota can say is that it was their intent to design the system to prevent that, and to the best of their knowledge that is how the system works… except in this case it didn’t work– unless the guy is simply lying or insane.

If the brake is pressed at the same time as the gas pedal, power to the engine will be reduced just as if the gas pedal had been released, the automaker said.

Unless, of course, there’s a malfunction of the system, which is exactly the issue under consideration.

During driving tests on Sikes’ Prius and on an identical 2008 Prius, the system operated as expected, according the report, preventing the car from pushing forward while braking.

“The system operating as expected” is not actually possible to determine, because they can’t see inside all the software and hardware to detect that every bit and electron is in the right place.

What they can say is that they detected no problems. Problems may be there, they just did not detect them.

If the visible, detectable problems we want to see are triggered by a transient event, such as a specific combination of foot presses, or perhaps there’s a two microsecond window of opportunity for two software events to happen simultaneously (such as have dogged the Mars rover missions ), then of course driving it around a parking lot is probably not going to reproduce the problem.

It is also possible that part of the problem involves a piece of physical equipment that was lodged or worn in a certain way at a specific temperature, and that condition no longer exists on the car in question.

When we try to reproduce problems, we often have to guess at the causes, and we may guess wrong.

If I were Toyota, I would treat this like an epidemiology problem. You interview people and make a list of absolutely everything that was going on. Did they have a cell phone? What kind? Where was it in the car? Where they using the cup holder? What drink was in the cup? Hot or cold? Was the air conditioner on or off? What was the setting?

Then you put all the data into a database and mine it for patterns.

Investigators are extremely meticulous when taking apart a car in a case like this, said Ed Higgins, a Michigan personal injury attorney who has been involved in automobile defect cases. They are aware their work will be subject to intense scrutiny, so they measure and document everything, he said.

That kind of care takes a lot of time. But it hasn’t been very long since the incident occurred. Have they also taken the software apart? Have they comprehensively reviewed the code? I seriously doubt that.

“I would think that any mechanical defect that would have allowed something to happen that otherwise could not have happened would have stood out like a sore thumb,” he said.

Unless it’s a transient interaction between a mechanical defect and an invisible state within the software.

The car also did not show damage consistent with the engine having been running at full throttle while the brakes were on, according to the report.

That suggests the brakes weren’t on, but not that Sikes wasn’t pushing on the brakes.

“Toyota engineers believe that it would be extremely difficult for the Prius to be driven at a continuous high speed with more than light brake-pedal pressure, and that the assertion that the vehicle could not be stopped with the brakes is fundamentally inconsistent with basic vehicle design and the investigation observations,” Toyota said in a statement.

Again, this all assumes normal circumstances and no transient failures. For the purposes of investigation, that belief is irrelevant.

It is already fundamentally inconsistent with the design of the product that ANY failure could occur. We’ve crossed that bridge, guys.

Remember, when flight 427 crashed, Boeing maintained for years that their rudder mechanism could not possibly have failed– until a new form of failure was discovered (“thermal shock”) and the specific failure reproduced in that very rudder assembly.

The car’s front brakes showed significant wear and overheating, Toyota said. That kind of wear and heat would be consistent with the brakes being lightly applied over a long period of time, executives said at the press event.

Data from on-board computers indicated that Sikes had applied the brakes, to some degree, at least 250 times during the 23 mile incident, Toyota executives said, and that the brakes worked normally each time.

Ooh, I love log files. I wonder what other patterns they can mine from that log file?

If the computers indicate that Sikes had applied the brakes, that shows they were getting some kind of signal from the brake mechanism, but not necessarily the correct signal. Therefore saying “the brakes worked normally each time” is completely unwarranted. Part of the system may have been working normally while another part was going haywire. There’s not way to tell after the fact, because “working normally” is not a detectable condition that gets logged in computer.

Every time you experience a problem in your software, your software, on some level, thinks it is doing the right thing. Software doesn’t “know” it’s misbehaving. It just does what it is told.

Edmunds has independently tested Prius cars similar to Sikes’ and confirmed that the engine would stay engaged if the brakes were only pressed lightly, but not hard enough to actually stop or slow the car, said Dan Edmunds, head of auto testing for the automotive Web site Edmunds.com.

He says “would”, but he should say “would, assuming that there is nothing wrong with the car that would cause it not to”

“If you’re just riding the brakes, it will ride the brakes,” he said.

“These findings certainly raise new questions surrounding the veracity of the sequence of events that has been reported by Mr. Sikes,” said Kurt Bardella, spokesman for Rep. Darrell Issa, R-Calif., and ranking member of the committee.

Sikes’ attorney, John Gomez, denied that the report proves his client was wrong about what happened to his car.

“The notion that they weren’t able to replicate it in this particular case tells us nothing,” he said. “They haven’t been able to replicate a single one of these.”

That’s right. Also, learn the phrase “transient failure mode” and press that point. There are plenty of examples in space missions and airliners of such failures.

Sikes has no plans to sue Toyota, Gomez said.

Gomez is also representing the family of Mark Saylor, a California Highway Patrolman who was killed, along with members of his family in a Lexus sedan that accelerated out of control. A preliminary investigation has found that the accelerator pedal in that car probably became trapped on an all-weather floor mat that had been incorrectly installed in the vehicle.

Toyota has issued a recall for several models, including Sikes’ Prius, to address possible floor mat entrapment. Sikes’ floor mat was not interfering with the accelerator, investigators found, and there were no signs the pedal had become stuck in any way, according to the report.

The investigators findings “suggest that there should be further examination of Mr. Sikes account of the events of March 8,” Toyota said in its statement.

Toyota spokesman Mike Michels also took issue with media coverage of the Sikes incident. Journalists sensationalized an admittedly dramatic event, he said, but the public would have been better served had reporters waited for all the facts.

“We need to let investigations take their course,” he said.

Yes indeed. And this investigation has not done that. Be careful what you wish for, Mr. Michels.

10 thoughts on “Toyota Story Analysis

  1. Excellent points, James; I agree with your analysis of the situation. Thanks for giving a perspective that needed to be given.

    I have one point to question, though. You say:

    I don’t know much about the design of a Prius, except that my understanding is that there are no physical cable linkages. It’s software driven. When you press the brakes you are essentially double clicking on the “brake” icon with your foot mouse, hoping that the operating system agrees to apply the real brakes. A Prius is basically a video game console connected to a car. You drive the console, not the car.

    and:

    Eventually, a California Highway Patrol officer was able to catch up to Sikes and used the patrol car’s public address system to instruct Sikes to apply the brakes and the emergency brake at the same time. That tactic worked, and he was able to stop the car.

    I’d like to know if Sikes had tried the emergency brake alone, before this. Had he tried both at once before this.

    I can understand if he did not try both at once, because a normal driver would think if they don’t work individually, why would they suddenly work together? But in a drive-by-wire system, everything is mediated by software, and software can get into strange states. It’s technically plausible that only with both brake controls activated the software could be bounced out of whatever strange trance it got into.

    However, I’m not sure that’s entirely true. While the regular brakes may certainly be (and probably are) controlled by software, so as not to harm the engine, it’s my understanding that the emergency brake of a Prius works just like the emergency brake of any other car; that is, it’s a physical mechanism.

    Having driven a Prius several times myself, though I haven’t examined it in detail, I’ve experienced the ratcheting/clicking sound and haptic feedback when engaging parking brake that all parking brakes have, and when releasing it, it still feels like a physical mechanism. I’m not inclined to think that the emergency brake is applied through software and remains applied when the car is completely turned off. I’d also be surprised if there were no single, purely physical fallback braking mechanism and the Prius instead relied solely on software for its braking; that sounds like an accident waiting to happen (even more so than latent and elusive software bugs).

    So this leads me to question not (only) the order in which these things were tried, but the accuracy of the statement that applying both of the braking mechanisms at the same time was the actual solution. Thoughts?

    [James’ Reply: I want to know more about this. I Googled a bit and didn’t find the specs right offhand. My Dad has a Prius, so I’ll ask him.]

  2. James, I was shocked as well that Toyota can suggest repeating the bug report didn’t reproduce the problem in 2 hours of testing. I’ve spent more time than that trying to isolate bugs by repeating the reported bug steps, finding out information, asking more questions, trying a different set of input steps, finding out information, etc. before finally discovering the sequence that reproduced the bug.

    In the AP article it stated “In Sikes’ case, Toyota said it found he rapidly pressed the gas and brakes back and forth 250 times, the maximum amount of data that the car’s self-diagnostic system can collect.” So the log maxed out at 250. Could it have been hundreds or thousands more instances of alternating pedal presses? This sounds like a potential infinite loop problem. Perhaps Mr. Sykes sequence of gas/brake “clicks” put him in a loop.

    If I was a tester assigned to investigating this bug report I would want to know:

    Were these the only items in the log file? Were they at regular or irregular intervals and how long between events i.e. do the events look more like a computer produced them or human?

    Additionally, is there different logic for braking depending on speed and force of the pedal press? I would guess there is. Perhaps there is a bug only at certain speeds at boundary conditions.

    It will be interesting to follow this story in the coming weeks

  3. Hi James,

    interesting. Wonder if the lawyer will be calling you.

    This is the age-old question again. What do you do with non repeatable defects? I’d say in this case investigate but be aware there might be no answer. Wait for more cases. That migt sound cruel but:

    a) Compared to accident stats for the Prius this is not even a blip so relevance can be questioned
    b) just imagine they introduce a fix. And that they fixed the wrong thing (they can’t test it because it is not repeatable). Isn’t then the danger of damage even higher? Should not be erred on the side of less risk? Like the aviation guys do?

    There is a whole world of uncomfortable business and risk decisions here that are not really customer compatible but reality nonetheless. I have the distinct feeling someone is trying to get Toyota. They are finally vulnerable and the media is exploiting that out of proportion. So go figure.

    Anyway I still liked your disection. Really good example of staying objective.

    Thanks
    Oliver

    [James’ Reply: One of my posts from long ago, here is all about reproducing intermittent bugs.

    This has become such a large PR problem for them that I’m sure they want to find and fix the problem just on general principles.]

  4. Good post man. And regardless of what the reality is and how it all falls out, this really brings the issue of software quality and it’s true importance to the forefront. While airplanes and rockets falling out of the sky are horrible, they are few, far in between, and “happen to other people”. But Prius’s screaming uncontrollably down the highway are you and your friends. It is real, common, and at the moment apparently consistent enough to have everyone’s attention. We can only hope that this gets and stays big enough long enough to force companies to really begin to take seriously the commitment to proper risk assessment in software implementation. Having to reboot your laptop because your browser crashed is annoying. Mass carnage on the highways is a tad more serious.

    David

  5. Just because it’s a physical mechanism in the emergency brake doesn’t mean there isn’t a signal generated and sent to the onboard computer. What, do you think, turns on the light on the dash that indicates the e-brake’s position? 😉

  6. Fascinating article.

    I was struck by the fact that [He said dispatchers tried to talk him through ways to stop the car, but nothing helped.] so can I assume that they didn’t suggest to try the break pedal and the emergency break at the same time? I’d love to hear that call…

  7. Not having knowledge about the car, couldn’t the driver just turn off the engine?
    Any got any knowledge about this?

    [James’ Reply: This is from my dad…

    I took the Prius up to the speed limit on a long straight main road (40 mph), moved the shift lever to N and held it there for two to four seconds.

    The gearbox (or Drive-O-Matic or whatever’s inside) went into neutral and stayed there till the car stopped (the brakes stopped the car without undue pressure). It would not shift into any other mode from N while the car was rolling with the system energized…select Drive and it simply beeped a couple of times and remained in N. Stopped at the side of the road, I shut the system off, pressed it ON again a few seconds later and all worked normally.

    To shut the system off while underway at about 10 mph, I had to hold the start button down in the OFF position for three to five seconds (unlike shutting it off while stopped, when a one-second touch kills the displays), at which the automobile went dead (at this speed the brakes seemed to work normally). It would not restart while the wheels were rolling. After it was stopped, it restarted normally with the ON button.

    I have no idea if this is what the Prius is designed to do…this is what our 2008 Prius did while moving straight ahead under relatively low-speed, cool-weather, dry-road, low-humidity conditions with a cellphone present and ON but not activated in the vehicle; with the radio off, the lights off, turn signals off, emergency flashers off, wipers off, heating system on LOW, windshield heater ON, doors and windows closed, no passengers in the car, one third of a tank of fuel on board, and the “Maint Req’d” light illuminated probably because it was programmed to come on the instant the mileage hit some magic number for warrantee service and not because maintenance is in fact required.
    ]

  8. Toyota is no doubt suffering in the court of public opinion. Their perceived lack of concern for their customers will have lasting affects. But Ford Motors comes in 2nd with over 50+ “Sudden Unintended Acceleration” cases. My Pontiac Car had a bad anti lock brake control unit so GM has also had serious recalls this year. I looked on http://www.carpedalrecall.com and found the recall info and local dealership listing; my co Worker had a pedal recall on his ford truck so just look out .

  9. I really liked your post. I can’t believe they are really convinced there are no problems.
    And if I were in their place, I would at least not declare that the driver story is inconsistent…

    Because when something like this happens (http://www.straitstimes.com/BreakingNews/Money/Story/STIStory_503200.html – this is about a Norvegian who saw his Prius accelerate), they will look like fools (if not like liers 🙂 ).

    What is curious is that the Norvegian Toyota say exactly the opposite as US Toyota, they say that the driver story “The (driver’s) description strengthens our conviction that our client was in a situation where his vehicle accelerated in an uncontrollable manner”.
    Now, I wonder what will US Toyota say about this :).

    I searched for more information on this, but I could not find the whole Toyota statement.

Leave a Reply

Your email address will not be published. Required fields are marked *