On the other hand, though, (and I know nothing about planes) would you then have done the manual checks too? Or only done manual checks if the automated tests had flagged something. I try for the former, but am probably guilty of the latter too often for comfort.

A very good and timely reminder to keep alert!

[James' Reply: A sapient pilot is always needed. Automation is fine as long as you don't mind when it fails. We have lots of automation on our cars, for instance, but failure there generally leads to being stuck at the side of the road. Automation failure on aircraft is considerably more risky. That's why we pilots are constantly watching and checking and cross-checking.]

James, I really liked your statement above. Since years you seem like the one of the few serious voices advocating automation WHEN it makes sense – and not just everywhere it might. The ROI in automation is indeed an issue and should be taken more seriously. Everybody can sell automation – it sounds so nice specially to those not in the know – and the dream of the magic button.

Funny how even after James gave, in his story, explicit critical criteria that pointed to the problem—color, taste, smell—they still don’t appear on the list.

The argument is simple: unlike humans, automation doesn’t have expectations. It is capable of detecting matches or mismatches based on the expectations we feed it. Humans, by contrast, can identify expectations prospectively, in advance of an observation (“If this were to come to pass, it would indicate a problem”); on the fly, during an observation (“Hey… I didn’t expect that; that could be a problem”); and retrospectively (“Now that I’ve seen the problem, I can figure out how I might have detected it” or “Now that I have a better model, I realize that I could have tested for this, even though a problem hasn’t manifested itself yet.”).

Just light the red lamp for a pilot if a parameter changed to a value outside of defined range.

One indicator of a problem with any analysis is the appearance of the word “just”. Yes: lighting a red lamp is a simple enough thing. Knowing when to light it and when not to light it is the complicated bit. Interpreting it is also complicated. Here’s an example of a one-bit communications protocol: someone honks the horn at you, as you’re driving. How would one interpret accurately that without a walloping dose of context to go along with it?

—Michael B.

This is the lesson to think about…

“Crash pilots given conflicting orders”
http://news.bbc.co.uk/2/hi/europe/2115425.stm

[quote]
German investigators have revealed that the pilots of the Russian airliner involved in last week’s mid-air collision with a cargo plane received contradictory instructions seconds before the crash.
They said voice recorders recovered from both aircraft showed Swiss air traffic controllers told the Russian pilots to descend, while the on-board warning system instructed them to climb.

All 69 people – including 45 schoolchildren – aboard the Russian Tu-154, and two crew members on the Boeing 757 were killed when the two aircraft collided at 35,000 feet (10,500 metres) over the German-Swiss border.

According to German authorities, cockpit warning systems told the Tu-154 to climb and the cargo jet to descend, just 45 seconds before the collision. But voice recorders reveal that one second later, Zurich air traffic controllers told the Russian pilots to descend. The Russian crew did not respond, so the Zurich control tower repeated the order 14 seconds later, investigators say.

The Russian plane responded and the two aircraft collided 30 seconds later.

Although the aircraft were flying over Germany at the time, they were under the control of the Swiss air traffic control body, Skyguide.

[/quote]

Thank you,
Albert Gareev

[James' Reply: It is customary to call an example like this a lesson. But really, it's not a lesson. This is a story. It becomes a lesson when we interpret it and make it a part of ourselves. My interpretation of this, I suspect, is different from your interpretation.

I see this as a lesson about the automation bias and the need for vigorous training in order to pilot aircraft safely.]

Great story today. Your one-liners such as:
“Unless you landed at the rainbow lake on Unicorn Planet, there may be a hydraulic leak in there.” or
\In that way, we are like secret service agents, trained to deal with known threats and to have a good chance to discern fresh and unusual threats, too.\
makes me smile all day in office.
Back to testing
Cheers

“The question “Can I explain everything that I see?” is a good heuristic to keep in mind while testing.”
I find this heuristic to be valuable. In the past, I’ve seen an anomaly, say, “yesterday the program wasn’t sending me emails, but today it is.” It’s insanely easy to say “Well, it’s working now, so it passes, and that’s ok.” I always try and make a point now of asking “why was it not working yesterday, but today it is?” rather than just being content that it now works. The investigative path often leads to very interesting information, even if you never end up explaining the anomaly.

Hi James,

You suddenly decided to pump the float (the action wasn’t really required) and discovered an issue that could have become a serious problem. That is a typical situation in testing but I don’t see why you [hypothetically] have problems imagining automation of float pump tests that would find unexpected/undefined problem. Below is the simplified illustration.

Layer One: Detection
I’m sure you’ve heard of Boolean algebra and I’m 100% confident you practically used elements of it as you did some programming.

[James' Reply: This is how you're going to start your argument? Okay. Why yes, Albert, I have heard of Boolean algebra. It came in handy when I was writing machine code for those video games I wrote for the Apple II, Commodore64, PC, and Amiga, back in the eighties. And indeed it's essential in writing the test tools and automation that I create, today. I sure hope the point you are about to make is better informed about computer science than it is about my technical background...]

The main rule that you would need to use here is “FALSE = NOT TRUE”
Implementation would require heuristic too: instead of looking for problems automatic unit must look after the object’s (float, pump, and peripherals) state. If object’s state is deviated from the base one – that is an indicator of [possible] problem. State should be defined by a sufficient set of criteria: amount of water in a float, chemical activity of water (pH, electro-conductivity, etc), vibration, pump performance, – as much as required.

[James' Reply: This fails in two ways to address the issue I raised, both of which you need to understand if you want to get good at test automation:

There are certainly situations we both could cite where we would think that any change to a particular variable is highly likely to be bad. However, in my experience working with complex dynamic systems and complex testing thereof, those situations are not commonplace. Frequently it requires clever coding to mince our way around things that are fully expected to change. The more you write such code, the more expensive your automation becomes.]

//Note. That already could be sufficient enough, depending on context. Just light the red lamp for a pilot if a parameter changed to a value outside of defined range. However, we may end up with too many lamps or red lights indicating low threat that could have been ignored if checked by a human being. To avoid that problem an automatic checking unit should become a little smarter on its own and do evaluation.//

[James' Reply: In the cockpit, I have a number of instruments that can indicate potential trouble, but just that indication-- which is all automation can do-- is not enough for excellent flying (or testing). There is no "everything is okay" detector on the plane. Nor can there be. The designers have anticipated various specific kinds of problems and I have detectors for those (a stall warning tone, for instance). But on top of all that-- and here is the point I'm trying to make-- my sapience must interpolate and extrapolate; generate ideas and filter them. The automation can help, but it cannot replace sapience.]

Layer Two: Evaluation
Boolean algebra is not sufficient enough here. Fuzzy Logic methods (or Artificial Neural Network for more complicated cases) are needed to be used.

[James' Reply: I would argue that a natural neural network is needed.]

From the context of your story: small amount of water in a float isn’t a problem itself; a little salty water is also a valid case. That is, “true” or “false” state of event doesn’t give basis for evaluation. It’s combination of events and weight of each event that matter.
Examples.
Little water + high electro-conductivity = alert
Little water + high/low pH = alert

[James' Reply: What is the false positive rate for that and wouldn't it be easier just to taste the water? This also demonstrates my point about pink water: I don't see anything here that detects pinkness. Here's some homework for you. At minute 27 of this video, this doctor explains how a particular form of automation has made medical testing WORSE.]

//Note. It should be seen pretty clear now that implementation of such a solution isn’t a quick patch. Therefore it should be reasonable and affordable.//

From the safety and security stand points it could be reasonable even for a light aircraft. Economically it could be affordable if it’s sold along with 100 of planes. Or 100,000. Or one large plane.

[James' Reply: But Albert, your very example suggests that it's intractable, because you are either unwilling or unable to supply me with a complete list of things to check for and a complete design of the equipment to check for it. Oh, and how much will this equipment weigh? How expensive is it to test and maintain? Simplicity and lightness is important in designing light aircraft.

You haven't addressed these concerns, nor would I expect you to, because my argument is that it's ridiculously hard to do so, and not at all worth doing compared to training the pilot to handle these things himself.]

It also may lead to what I wouldn’t want to happen but what could happen and happens anyway.
For example – replacing a highly qualified mechanic with a combination “automatic unit” + “2-week trainee”. It could be because it’s too hard to get or too long to grow a professional, or because it’s much cheaper to employ 10 surrogates. Or in an organization with extremely high turnover of staff. That used to be only army in action but now seems to be any big corporation.
This approach has come and stayed, and it looks very attractive in accounting books. What is not visible in balance sheets that professional armed with automation can bring performance to new levels of excellence while amateurish workers even with automation have problems performing just as good.

So my point is that instead of trying to stop the progress maybe better to walk ahead of it?

[James' Reply: What do you mean progress? I'm not standing in the way of any progress. But I would say that it is not progress to seek to do with automation expensively and poorly what can be done far better and far less expensively by a skilled tester.]

(Remember, Mars Pathfinder over-succeeded in 1997, working automatically, without direct radio guidance)

[James' Reply: Yes, when I taught my testing class at JPL and saw the new rover under construction. That's wonderful and extremely expensive technology. It also had a critical fault in it that almost ended the mission on the second day!

Dude, I'm not opposed to test automation. I'm opposed to wild irresponsible claims made about test automation. This is based on extensive experience with test automation that I have, as a programmer and as a tester and as a consultant who has written reports about test automation failures.]

Thank you,
Albert Gareev
http://automation-beyond.com/

I was wondering if the fact that he was your Dad affected the way you approached your ‘testing’? You mention that you were ‘determined’ to find something wrong with the plane. Do you think this is typical of your approach to life or is there a natural competition between you and your Dad?

Makes me think that perhaps who we work with affects how we test too!

Anne-Marie

[James' Reply: It doesn't occur to me to compete with him. I'm on the same team. I want to live up to the standard he embodies.]