Interesting problem, testing a clock (or, if you like, a pulsar) that you cannot interact with. One question I have is “test it against what, exactly?”. On the surface, you would just take a (purportedly) accurate timepiece and verify that the clock’s output (movement of hands) is accurate with relation to your device.

Unfortunately, any clock you use for verification is just another instance of the very thing you’re testing. Who can say that it is accurate? All it leaves us with is that we can verify with a degree of certainty that the hands (or, indeed numerals) are moving (or changing). We cannot accurately measure whether or not they are moving or changing at the correct rate.

Applying the same reasoning to the testing of a pulsar, the whole exercise would seem to be topsy-turvy. You would be better testing the operation of your timepiece against the output from the pulsar. Although, proving that would be difficult since we have had to use the same flawed measurement devices (however accurate we regard them) to measure the spin rate of the pulsar. Knotty one, that.

We also have to remember that hours, minutes, seconds et al are something we have made up to try to encompass the concept of ‘time’ and it’s value to us. It was never necessary until the church started trying to have regular services. Additionally, since time and space are so securely linked and we know that gravity affects the ‘shape’ of space, the proximity to a gravitational body would affect the output.

I guess it’s one of those questions. No real answer but that’s OK, it’s the path you take that’s important.

One last thing, you give the example of a pulsar 100 light years distant and suggest that the distance makes it impossible to interact with. Well, every object that has mass (ie. all of the particles that make up ‘us’) have a gravitational effect. Gravity is a weak force but it has incredible (and arguably infinite) reach, meaning that we can’t NOT interact with it, however feebly. Just being there has a miniscule but very real effect.

[James' Reply: Very nice analysis. Have I met you? I'm not surprised that you work for Test Partners.]

I hope we meet the next time I'm in Finland (which would be the first time).]

[James' Reply: Yeah, that is testing it without giving it input.]

Confirm that an initial input was to start or wind the clock. Ask whether any physical stimuli to the clock case was within the scope of testing. Confirm time zone. Monitor accuracy over a period of time, and ask over what period of time you will be allowed or expected to test the clock.

[James' Reply: You can try to confirm as you please, but you can't give it any input.]

BTW, I come from an Electrical Engineering and hardware test background, so I am more comfortable with the IEEE definition. However, I can definitely understand your desire to use a different definition that you feel is more comprehensive and usable.

[James' Reply: I come from a science and philosophy background, so I like definitions that stand up to critical scrutiny. I think you should want that, too. Look, you can't give the clock input. You seem to want to include all conceivable past input provided by anything or anyone in your "test case." Aren't you working a bit hard to maintain your definition? You might as well call all of history a test case and be done with it.

Besides, I can construct a situation where the product takes no input any time in its history. It's just a mechanism that performs. It seems to me it is still possible to test such a thing.]

Finally, you seem to have a very different style than I do. However, I will say two things: Test is almost by definition challenging the correctness/completeness/speed/etc. of an object. And the path I take to accepting an idea or point of view generally involves challenging it.

[James' Reply: I'm glad to hear that. That's crucial to the testing spirit. If you and I were in the meeting where the IEEE guys were debating the definition-- wait, I'm sorry, I've been in IEEE standards meetings (For IEEE-829). There is no serious debate. No one has the stomach for it.]

Yes, it is possible to test the clock when you can not interact with the clock except to see it. I believe many readers have already suggested ways to test the clock without handling it physically. In fact, it is even possible to test the clock without the clock even existing.

Testing the clock is as much about the clock as it is about the tester of the clock. The quality of the clock is determined (measured) by the tester (end user) based upon certain criteria or preconceived expectations.

[James' Reply: I agree. What I hope to do when I present this puzzle to a student is to drive an examination of what the word "testing" might mean, and also to help them see that although some testing is not possible in this situation, other testing may be.]

Coming back to the non-existing clock, I’m going to ask the help of my 5-year old daughter to test your clock that doesn’t exist yet. I told her all about this wonderful new clock that lives in a glass jar. I further told her that my friend is building a factory to build this glass jar clocks but the first one will only be available in 2 years. She only asked me one question: “Daddy, is it pink?”. I had to confess, sorry, it only comes out in black. She decided that she doesn’t want one, since she loves pink.

[James' Reply: Nice example!]

In the example above you may translate it to a market survey.

In conclusion, testing is always possible, but there might be limitations to the amount or kind of testing that may be performed. Anticipating and compensating for possible violations of expectation is key.

[James' Reply: Thank you for providing an excellent example of what it looks like to go through the sort of puzzles that I believe help us be better applied philosophers of epistemology (a fancy way of saying "tester").]

It is an interesting logical clock puzzle you have put forth.
I’m going to take you up on it, but only if you meet my own challenge to stick to your definition of the clock and restrictions surrounding it. Any solution is invalid if you change the base definition. If I misunderstood your base definition I will concede defeat, but if my solution applies, you will have to concede. That is the rules.

[James' Reply: There is no ultimate right or wrong solution in my puzzles. The point is how you think them through. That's the learning we can use.

This particularly is not a logic puzzle (not like a chess puzzle, for instance), because there is nothing well-defined about it. To solve it is not a matter of logic, but definitely a matter of inquiry, modeling, and reasoning.]

Your puzzle is defined as:
“James: So, let’s imagine an acrylic box. Inside the box runs a clock. You can see the clock face. You can’t interact with the clock (except to look at it in normal light). Can the clock be tested?”

Solution:
I would ask my friend Joe to open the acrylic box by using solvents such as di- or trichloromethane to remove the acrylic.
Then I will instruct Joe how to handle and perform any test deemed necessary.
At no point in time will (I) (see: YOU can’t interact with the clock) be directly interacting with the clock.

Q.E.D.

[James' Reply: Okay, you've done something that all good testers ought to be able to do when confronted by an obstacle-- try to overcome it. That's good.

My intent in posing the puzzle is to create a situation where no one can interact with the clock. You can see it. That's all. Is it possible to test in that situation? Hence, your solution is not "wrong", but I'd like to hear more.]

I read about “Meeta Prakash” on this blog post and came to know that we work for the same company.

Just waiting for more of your blog posts!

But I suppose in that specific case, we would say that a scientist is using the pulsar as test data to test a theory he has about pulsars, because the pulsar itself would not be thought of as "defective" if it didn't match the theory.]

If I’m observing a pulsar from 100 light years away, everything is forced into an observation-only mode. AND if I know nothing of pulsars, then I start out like the blind men investigating the elephant. My initial assertions may be flawed, but as I continue to observe and gather data I have continued opportunities to refine or rewrite my hypothesis. I would say that this is where astronomers, cosmologists and the like differ from other scientists – they don’t always get to benefit from direct interaction or manipulation of the items under study.

[James' Reply: Yeah.]

Taking what little I’ve learned and forming it into a model, whether concrete and physical or a simulation on a computer, I can start to interact in a more physical manner and applying direct inputs, different conditions, etc. A good friend of mine, who worked in the aerospace industry, has often commented on how they built satellite and rocket systems that had all of the components tested without actually putting something into space. This form of testing relied heavily on models and simulators to determine things like flight characteristics, system interfaces, etc. Not saying that the aerospace industry is perfect, but they have managed to put many kinds of complex objects in the air and into orbit, successfully, with this kind of approach to testing.

[James' Reply: It's a great idea. I think it can be applied in software testing, too.]

I think we test, in many cases, without knowing what the configuration is or having the ability to change it. I can think of times where I have had to do large scale integrations of systems, many of which I didn’t own OR have control of. In which case, it didn’t limit me from using them to evaluate other systems that I did have control. What it did, however, was force me into a position of a risk-based decision – A) do no testing or evaluation of any system because I cannot control one or B) call out the risk and evaluate what I do have under my control. Either position has risks, but option B gives me some opportunity to learn about the systems, even though it may not be complete.

[James' Reply: There's a difference between not knowing the complete configuration and not knowing any of it. Obviously, you know that the system is "running" as opposed to "uninstalled." Or at least you THINK you know that. Otherwise you wouldn't bother to test at all.]

1. turn on/off the room lights – maybe it’s solar powered – Can I cause the batteries to drain/charge? Can I force the display to “sleep”? Can I see if the display is lit/backlit in the dark? Can I tell if the display changes gets brighter/dimmer with lights on/off?

2. inducing radio frequencies – maybe it’s “atomic” powered – can I interefere with the radio reception that controls the settings? Can I inject changes with radio waves of my own?

3. magnetism – can I induce a magnetic field that interrupts the clocks mechanism? If it’s digital, possible? How about mechanical?

4. simple comparison – can I compare the clock under the glass with a similar make and model that I can manipulate/touch/feel/shake? (maybe using the clock in the glass as a control?) How about comparing it with my wristwatch or cellphone?

5. temperature – if I change the temperature in the room (hotter/cooler) can I detect any changes in the clock?

6. pressure – if I change the pressure in the room can I detect any changes in the clock?

7. back to the comparison – can I go home and find a similar make and model that I can interact with?

[James' Reply: Good ideas. For those ideas that involve subjecting the clock to different conditions, I would praise your ingenuity and agree that when confronted with limitation we should try to find a way around them. But then I would re-assert the notion that we can't influence the clock. If necessary I would say "What if the clock were a pulsar, 100 light years away? Can that be tested in less than 200 years?"

Testing a model of the clock is especially clever. I would ask you how we could use what we learned to test the original clock?]

1. “…the point I’m trying to get at is how to test something that you CAN’T interact with.”

2. “The glass dome, or box, represents condition limiting your ability to test it.”

[James' Reply: What's different about these? I don't see anything contradictory, here. Perhaps we have difference definitions of interaction. See below...]

There’s a reason the clock’s container is glass/acrylic: I have to have some sensory input. Only if I have sensory input that can be traced to the object do I have interaction with the object under test.

[James' reply: I would say you don't have interaction. You have the ability to observe. An astronomer can observe a star, but we would not say he is interacting with it.]

Sensory input is to some degree quantifiable, and can be compared to similar input from available oracles: this, to me, qualifies as a test.

[James' Reply: I agree. That's part of solving the puzzle. You don't need to interact with something in order to test it in SOME way. You may need to interact for some kinds of tests, but not all kinds.]

A conditional limitation is not a complete absence of interaction; that would be an unconditional limitation [or an unlimited condition?]. I contend that you cannot test something for which you have no knowledge, no awareness, no sensory stimulation.

Give me sensory input and an oracle, and I can test.

[James' Reply: I concur. Good job. The point of the exercise is to pose what seems to be a dilemma, and then get the tester to refine his idea of testing to the point where he can see his way through it. You have just done so.]

[James' Reply: Part of the exercise is for the student to ask questions necessary to clarify that. Just as you are doing.]

Mentally, I envisioned your acrylic box with the clock inside: a large, six-sided, ticking die. One of the first tests I imagined was measuring whether the clock would operate identically regardless of the orientation of the box. Valid test: I cannot interact with the clock, but nothing I read in your requirements limited my interaction with the box.

[James' Reply: I would reply that you cannot approach the clock or interact with it, physically, in any other way than to look at it from a short distance away.]

Returning to your blog, some of my test cases are irrelevant because the item under test has morphed from an acrylic box to a glass dome. With one product, measuring the clock’s accuracy while it is upside down is a legitimate test case; with the other product, it’s an ID-10T error.

[James' Reply: It's nice that you are thinking of ways around the restrictions. I would reward you for that, and then re-emphasize the restrictions, because the point I'm getting at is how to test something that you CAN'T interact with. Of course, a secondary point is that you ought to think of ways around any restrictions that inhibit good testing.]

Or, as you said to Michael in the transpection, “You can’t just make things up, my friend. It helps to have definite ideas about what words mean.” The product is NOT the clock, contrary to what you stated. The product is either the clock in the acrylic box, or the clock in the glass dome; either way, these are two different items which will have different tests.

[James' Reply: Let's say that the product is the clock. The glass dome, or box, represents condition limiting your ability to test it.]

As for the question, yes, not exactly asked that way, but very similar.. the question was.. (not exactly this, but I’m paraphrasing here) “Here is a clock, you can’t do anything to the clock (anything I assumed implied touching, moving, etc), how would you test the clock to make sure it keeps time accurately. (something to that affect)… it definitely threw me for a loop, but it was fun none the less trying to answer it.. I can’t remember exactly how I answered it… but I remember a lot of silence on the other end of the phone interview (I still was asked to come back for another round.. but didn’t take them up on it).

[James' Reply: Interesting. I hadn't heard about that. I guess it's a natural sort of example, though.]

[James' Reply: Which question? The clock? I made that one up. Did you really get that as a question at MS? Or are you talking about a different question?]

As to your post, I’ve never taken criticism personally, but I think many do.. it’s a shame really, since I never learned by following the lemmings off the cliff – I’ve learned fundamentally from being challenged and dropped on my head. If everyone told me that my approach or ideas were gospel, well.. then, I’d be …. king of the universe or something. A good discord is great for the soul.

Not really on this topic, I’m curious what your take is on BDD using tools like jBehave or Cucumber. I seems to me an early offshoot of Keyword Test Design (which is my baby). I’ve been experimenting a ton with it lately, and it really is some neat stuff. In many respects it offers some additional context to test design development that maybe keyword automation is lacking, but I’m curious how or if you’ve played with these tools/method and what your opinion is as to it’s ability to support using ET methods of test creation and it’s eventual execution automatically.

[James' Reply: Actually I find the whole concept troubling and misguided. Perhaps I have not seen it demonstrated properly, yet.

I have the strong impression that whoever wrote Cucumber is really excited about programming and really uninterested in testing.]

Just had the CITCON (Continuous integration & Testing Conference, http://citconf.com) in Wellington. It is an unconference and I put the topic up “Test Automation != Quality & != Testing”. This was probably the most visited session of the conf and the most hotly discussed. It digressed quickly to how we do testing today and where everyone was coming from.

1hr is certainly not much to discuss this topic but what was immediately apparent was:

1) There were at least 45 people absolutely passionate about testing
2) They were all in for the “fight” and knew how to
3) They all had good arguments (even if I didn’t agree with most)
4) Developers and testers do have different languages (misunderstandings were not immediately evident)

I think we all took away heaps of thoughts from this discussion and it made me proud that there is so much enthusiasm in NZ on this topic. Be it developer or tester.

I think there are now 45 people more that have realised that testing is not just as simple as some make it out to be, that it can be far more complex than anticipated and -most importantly- it IS open for debate.

All this just by prodding “it” with a stick(ie). So James the confrontational route really works and inspires. Even those that might not see the light the first time round might be back for more l8r on.

Oliver

“you can point to a list of test ideas, in bullet form, and call them test cases, just like real people at real companies already do, and not be committing a crime against terminology”

Nice! I want my team to test. I don’t want them to spend any more time than necessary writing test cases (or writing anything that isn’t absolutely necessary, for that matter).

In many cases, that means jotting down bullet-point test ideas, rather than paragraph-form prose. All I really ask is that they put their work products on our intranet site so that we can all share and learn from it.

I get some nervous feedback from newer members of my team (“what template should I use?”, “is that really all you want me to write?”), but eventually they learn that I trust them to do whatever works for them, as long as it works for the company as well.

“All this reminds me of Anna Allison.”

Thanks for the reminder. I think of Anna occasionally as well. We used to work together on the QA team of a now-defunct software company. While she was a person full of terrific ideas, she was always open to new ones.

The first task I was given when joining the company was to implement automated testing, and at the time, Anna was the company’s biggest skeptic concerning test automation. I knew I had cleared a major milestone when I was able to “convert” Anna, by helping her team begin to automate some of their test efforts.

Anna and I remained friends when the company went downhill and we each moved on, and we often had some great discussions about software and testing. We had lunch shortly before Sept 11.

I still miss her – our profession is poorer for her absence.

It’s like when you try to teach/talk to a colleague about a topic and the “why” and “what do you mean by that” questions come up.

It forces you to go deeper or re-assess your approach (either you haven’t got to the right level of detail/background yet or your approach is making an assumption about how the other person thinks/responds…)

So, either way it forces you to think, analyse what your own pre-conceptions are and understand more detail of what you’re trying to say. This self-evaluation is key.

For any thinking tester this is necessary to help their own learning and understanding about a topic – those are the tools that will help them ask better questions and even look for assumptions that others are making (whether it’s about a product feature, claim or bug.)

So, challenges per se are a “must” or a /good/ thing. Of course, how people respond to them is a more personal thing – sometimes it’s instant and sometimes they need their own space, go away and think about it. Ruminate, cogitate and then respond, maybe.

But it’s the thinking that’s important – and even showing your thinking.

A good while ago, when doing maths at school and college “showing your working” was actually quite important – it was an insight into your assumptions and map-making to come to a solution, idea, hypothesis, conjecture etc.

I try to re-use that idea – not only because it can help others understand my point or question – but I might re-visit the idea sometime later and then it’s a useful reminder about what my own assumptions were “at the time” – they might not hold true longer or been forgotten. Plus it helps to stop things getting lost in translation – even when you have the same language!

I’ll stop before this turns into a blog post on your comment page. Thanks for the post – it prompted me to think a bit about the topic – and why I think challenges are a good thing, but responding to challenges and “showing your thinking” are even better!