Comments on: How to Investigate Intermittent Problems http://www.satisfice.com/blog/archives/34 The Consulting Software Tester Fri, 03 Feb 2012 10:21:43 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Wade Wachs http://www.satisfice.com/blog/archives/34/comment-page-1#comment-262456 Wade Wachs Mon, 18 Oct 2010 20:10:23 +0000 http://12.165.213.55/blog/?p=34#comment-262456 It seems to me that what is being said is that the term 'intermittent' just means 'I don't know how to reproduce it yet'. <em>[James' Reply: Pretty much.] </em> If a bug exists with a certain set of inputs/states, it seems the more possible combinations of inputs and states the more possible outputs you have. <em> [James' Reply: Not necessarily. The possibility of outputs is unconnected to input. Consider a random number generator.]</em> Consider a simple circuit with a switch, power source, and light bulb. When the switch is turned on, the bulb illuminates. When the switch is turned off, the bulb goes dark. If we turn the switch on, and the bulb does not light up, we call this a bug, and we diagnose what we know, for example the power source is dead. This is not considered an 'intermittent' bug, we found the problem and can easily reproduce it. <em>[James' Reply: It's intermittent if it happens some times and not others.] </em> Now, consider a circuit with hundreds of switches, bulbs, resistors, power sources, capacitors, etc. all wired in parallel all with changing input and output as we are trying to test the effect of our switch on our light bulb. If changing our switch only illuminates the bulb some of the time, we call this an intermittent bug, when in reality it is a reproducible bug that we just haven't figured out how to do so yet. <em>[James' Reply: We call a bug intermittent based not on its own nature but on our view of it.] </em> Our white whale exists somewhere in the hundreds of other components that we weren't looking at in our test. Just because we don't know about all the inputs doesn't make the bug intermittent, it just means we haven't learned everything about it yet. <em>[James' Reply: It *is* intermittent-- to us.] </em> If we were to try to defend the term 'intermittent', we would have to somehow define the level of complexity before a bug can graduate from a reproducible bug to an intermittent bug. In the first example, if our power source (which is unknown to the tester) was a hamster running on a wheel with a generator attached, it would seem that intermittently the switch would turn the light on. But since we don't know the power source, we call this an intermittent bug even though it is seemingly simple system. How can we ever define the total set of our unknowns? We can't, therefore 'intermittent' simply becomes a way of saying 'there is something I don't know about the system that is affecting my results'. <em>[James' Reply: Yes.] </em> (I like where this is going, I'm thinking I will make this into a blog post for my own blog out of this. If you're interested in following this line of thought go check out wadewachs.com) <em>[James' Reply: I'm already following your blog.]</em> It seems to me that what is being said is that the term ‘intermittent’ just means ‘I don’t know how to reproduce it yet’.

[James' Reply: Pretty much.]

If a bug exists with a certain set of inputs/states, it seems the more possible combinations of inputs and states the more possible outputs you have.

[James' Reply: Not necessarily. The possibility of outputs is unconnected to input. Consider a random number generator.]

Consider a simple circuit with a switch, power source, and light bulb. When the switch is turned on, the bulb illuminates. When the switch is turned off, the bulb goes dark. If we turn the switch on, and the bulb does not light up, we call this a bug, and we diagnose what we know, for example the power source is dead. This is not considered an ‘intermittent’ bug, we found the problem and can easily reproduce it.

[James' Reply: It's intermittent if it happens some times and not others.]

Now, consider a circuit with hundreds of switches, bulbs, resistors, power sources, capacitors, etc. all wired in parallel all with changing input and output as we are trying to test the effect of our switch on our light bulb. If changing our switch only illuminates the bulb some of the time, we call this an intermittent bug, when in reality it is a reproducible bug that we just haven’t figured out how to do so yet.

[James' Reply: We call a bug intermittent based not on its own nature but on our view of it.]

Our white whale exists somewhere in the hundreds of other components that we weren’t looking at in our test. Just because we don’t know about all the inputs doesn’t make the bug intermittent, it just means we haven’t learned everything about it yet.

[James' Reply: It *is* intermittent-- to us.]

If we were to try to defend the term ‘intermittent’, we would have to somehow define the level of complexity before a bug can graduate from a reproducible bug to an intermittent bug. In the first example, if our power source (which is unknown to the tester) was a hamster running on a wheel with a generator attached, it would seem that intermittently the switch would turn the light on. But since we don’t know the power source, we call this an intermittent bug even though it is seemingly simple system. How can we ever define the total set of our unknowns? We can’t, therefore ‘intermittent’ simply becomes a way of saying ‘there is something I don’t know about the system that is affecting my results’.

[James' Reply: Yes.]

(I like where this is going, I’m thinking I will make this into a blog post for my own blog out of this. If you’re interested in following this line of thought go check out wadewachs.com)

[James' Reply: I'm already following your blog.]

]]> By: Gaurav http://www.satisfice.com/blog/archives/34/comment-page-1#comment-247160 Gaurav Wed, 10 Mar 2010 17:24:55 +0000 http://12.165.213.55/blog/?p=34#comment-247160 As per my understanding, it is easier (and cheaper) to find problems between two components rather than finding problems between more than two components. I gather, this is the basis of Pair Wise Testing and I have understand AllPairs, the same logic has been implemented. I was wondering, if you would be able to provide me some examples on how best to isolate defects between more than two components. (Assuming that such defects in some way maybe considered as intermittent) Regards Gaurav Pandey <em>[James' Reply: Can you present that as a more natural problem? What would it look like in real life? If you see something that is intermittent, you won't know which variables make it intermittent, or how many they are.]</em> As per my understanding, it is easier (and cheaper) to find problems between two components rather than finding problems between more than two components. I gather, this is the basis of Pair Wise Testing and I have understand AllPairs, the same logic has been implemented.

I was wondering, if you would be able to provide me some examples on how best to isolate defects between more than two components. (Assuming that such defects in some way maybe considered as intermittent)

Regards
Gaurav Pandey

[James' Reply: Can you present that as a more natural problem? What would it look like in real life?

If you see something that is intermittent, you won't know which variables make it intermittent, or how many they are.]

]]> By: chris http://www.satisfice.com/blog/archives/34/comment-page-1#comment-197669 chris Fri, 17 Jul 2009 00:47:53 +0000 http://12.165.213.55/blog/?p=34#comment-197669 The essence of intermittent bugs are of chaotic nature - where multiple (at least three) factors come into play. So there is no direct \cause\ for such bugs versus regular bugs. It is impossible for over 99% people to accept that there is no direct cause for the chaotic world. The famous three-body problem is a good start to understand the essence of intermittency. It is NEVER possible to simplify a true intermittent bug into a regular bug <em>[James' Reply: The whole notion of "cause" is artificial. You could as well say that anything that happens has an infinite number of causes. You're right, the world is chaotic. But I guess we could say that one reason a bug can be intermittent is because of a set of causes that are uncorrelated and unknown. But when I'm investigating a bug, some of its causes may be temporarily correlated or uncorrelated. There is no "true intermittence" because there is no way to be sure about the nature of the causes, and no way to prove that the causal structure won't change. In other words, give me an example of an intermittent bug, and I can give you one new fact that, if true, would reveal it to be not intermittent (or vice versa). Therefore, you might say "there is no way to be sure that a regular bug isn't secretly an intermittent bug, and no way to know that an intermittent bug isn't secretly a regular bug." Hence I call something an intermittent bug if, at the time I'm investigating it, it is intermittent. But I don't need to say it's "truly intermittent."]</em> The essence of intermittent bugs are of chaotic nature – where multiple (at least three) factors come into play. So there is no direct \cause\ for such bugs versus regular bugs. It is impossible for over 99% people to accept that there is no direct cause for the chaotic world. The famous three-body problem is a good start to understand the essence of intermittency.

It is NEVER possible to simplify a true intermittent bug into a regular bug

[James' Reply: The whole notion of "cause" is artificial. You could as well say that anything that happens has an infinite number of causes. You're right, the world is chaotic. But I guess we could say that one reason a bug can be intermittent is because of a set of causes that are uncorrelated and unknown.

But when I'm investigating a bug, some of its causes may be temporarily correlated or uncorrelated. There is no "true intermittence" because there is no way to be sure about the nature of the causes, and no way to prove that the causal structure won't change.

In other words, give me an example of an intermittent bug, and I can give you one new fact that, if true, would reveal it to be not intermittent (or vice versa). Therefore, you might say "there is no way to be sure that a regular bug isn't secretly an intermittent bug, and no way to know that an intermittent bug isn't secretly a regular bug." Hence I call something an intermittent bug if, at the time I'm investigating it, it is intermittent. But I don't need to say it's "truly intermittent."]

]]> By: Trevor Hopkinson http://www.satisfice.com/blog/archives/34/comment-page-1#comment-159325 Trevor Hopkinson Tue, 11 Nov 2008 16:28:13 +0000 http://12.165.213.55/blog/?p=34#comment-159325 But how to handle these intermuittent faults Appearance on test results reports - how to handle an 80% test pass rate what to do with bug reports raising - clearing - raising again Is it really a bug? - Any ideas welcome Trevor <em> [James' Reply: What's the problem? What is an 80% pass rate? Why is the bug report being raised and cleared?]</em> But how to handle these intermuittent faults

Appearance on test results reports – how to handle an 80% test pass rate
what to do with bug reports raising – clearing – raising again
Is it really a bug? –

Any ideas welcome

Trevor

[James' Reply: What's the problem? What is an 80% pass rate? Why is the bug report being raised and cleared?]

]]>