- Version
- Download 21538
- File Size 233.42 KB
- File Count 1
- Create Date 27 April 2014
- Last Updated 7 July 2021
Test Cases Are Not Testing
This is an article by me and Aaron Hodder for the debut issue of Testing Trapeze.
In this article, we explain why creating and performing test cases is not the same thing as doing testing. In fact, test cases are a poor basis for organizing a test process and no basis at all for measuring testing progress. Our industry's obsession with test cases must come to an end or testing as a field will be never be worthy of respect.
Attached Files
| File | Action |
|---|---|
| test-cases-are-not-testing.pdf | Download |
That was a very interesting read! I’m brand new to Software Testing and found your article while looking through r/SoftwareTesting on Reddit. I’m currently reading the textbook, A Friendly Guide to Software Testing, and just made it to the section on writing Test Cases and I’ve been considering the inherent problem of having two or more people read a test case in a different way.
Do you feel the way to combat that would be to have a clearer understanding of what the product is supposed to do and behave like? The chances of a test case being misread are lowered then, or is this misreading unavoidable?
I really enjoyed reading your thoughts on Test Cases and look forward to reading more articles on this site. Thank you for sharing.
[James’ Reply: The most important question is do you know how to test? After that, do you understand and accept the test strategy? After that, we can consider whether you understand a test case. But I don’t think misreading, as such, is much of an issue, unless you see a test case as a blind order sent to a blind tester. My advice is: don’t work that way. Testers should not routinely test from a position of ignorance and should not be dictated to by mysterious procedures written by shadowy people.]
Hi, this is an interesting read. You mentioned testing is an activity of knowing the performance of the product. By not writing test cases. Then how would you validate it? Are you indicating to go for more exploratory testing rather than taking a structured approach? I did not really get what solution you are providing and how to achieve that.
[James’ Reply: The answer to that question is in the article you are commenting upon. But to reiterate: validation is not a helpful word. What we do is test a product. We are looking for bugs in it. Testing is learning via experimentation. Of course this process is exploratory. If you aren’t exploring then you also aren’t testing. This is a structured process, by any reasonable definition of structured, so I don’t understand your point, there.]
All the arguments in this paper apply to test automation? Doesn’t it also apply to the best developers who don’t understand testing? (rhetorical question – answer is not needed).
Hi. I completely agree with the content of this article. However, on the software projects that I’ve worked on test cases were used to provide some metric of requirement coverage and (perhaps more importantly) to document the tester’s thoughts about a requirement (or the system) during testing (our test case documents mostly contained thoughts and ideas, the test cases were a minor part of the documents). I think of testing as everything that leads up to the writing of a ‘final’ set of the test cases.
[James’ Reply: You can use anything as a metric. For instance, you can save all the contents of the system logs that Windows keeps and track the rate at which they grow. You can keep track of every change to every file on the test system. You can count the number of HTTP requests you make to the back end. You CAN do these things, but I bet you don’t– because they are nearly useless as indicators of testing effort or value (although not completely useless). Similarly using test cases as a metric is also, at best, nearly useless, and at worst utterly misleading. That’s why I don’t use them as metrics and why no one else should, either.
Instead of discussing what desperate and clueless people might do, let’s discuss what is helpful and reasonable. Instead of using test cases as a unit of testing, use attention over time (i.e. chartered and timed test sessions). By mapping what you have been testing, how you have been testing, and how long you have been doing so, you can tell a compelling testing story that isn’t just numbers for the sake of it.]
Without writing test cases (or something similar to record what has been tested) I don’t know the project can be confident that the tester has at least made a good stab at testing the software. Do you know of alternative ways to record test progress?
[James’ Reply: Yes. Activity-based test management. One such method is Session-Based Test Management, which I developed for that purpose 20 years ago.]
Thanks James, Totally agree for all mentioned on your article, however I think that Test cases is a basic task for Testing work but the matter of discussion is how to write these test cases, one can see it is important to write a very detailed test cases, another one can see that it is enough to write high level test conditions, a third one say for my case it will be time consuming to write test case as it is very clear and only I can write an outline of testing work like only a titles for test cases, so for the freshers and new joiners, detailed test cases is required after that, after gaining the experience, I think it should be according to the situation you are facing, like if it is a common testing task thus you only need to write high level detailed test cases like test conditions only and on this case writing detailed test cases is only wasting time , and for a new or complicated testing task, you have to do testing analysis and write down very detailed test cases, otherwise it is a kind of lack of proper care and attention.
[James’ Reply: I’m not sure I understand what you are saying. If you are saying that test cases are an essential part of testing, I disagree. They are definitely not. The idea of test cases is completely invented. It’s optional. It’s not a natural truth about testing. I don’t agree that new testers must write test cases or even think about them. I do think any particular form of test documentation may be useful in some situation, and it helps to practice the various forms if you want to consider yourself to be an expert.]
This article reinforces my long held view that expecting a programmer to write his own unit tests is a waste of his programming skills. As you rightly say, programmers write programs while testers write tests. I have heard some programmers say that by writing their own unit tests they feel that they have become better programmers. This to me is a complete fallacy. As a programmer I am paid to write working code. Any time I spend writing unit tests is time NOT spent writing code and this must surely be a waste of my programming skills. Writing code and writing tests are different responsibilities, so expecting one person to carry out both responsibilities must surely be breaking the Single Responsibility Principle.
Hi James,
I like your articles and your way of thinking however my whole work experience in QA – no matter the project, industry, etc… – clearly says that in projects where there were no test cases were most ‘broken’ and without any quality control. I strongly believe – and this is based on own experience – that no written test cases is wrong and too many written test cases is equally wrong.
[James’ Reply: If you were a cook or a food critic and you wanted to talk about the quality of food, would you say “there aren’t enough written recipes?” I don’t think so. I think you would talk about the cooking technique or the quality of the ingredients or the look of the food. No one would simply say “the problem with this restaurant is there aren’t enough rules to follow!” Similarly, if I felt that a project would be improved by creating some written test cases, I wouldn’t express that by saying that the project “needed more test cases.” I would express that by talking about testing that needed to be done.
Testing doesn’t need test cases. Testing needs good testing. If that comes along with some test cases, fine. But test cases are NOT testing.]
In first case you easily forget to cover lots of important functionality (new and especially existing – regression testing). If you get sick – no other can substitute you. If you got well written test cases – at least basic package – other may test when you are not available – of course – it will not be as good as if you tested it – but it is something – and that makes a huge difference.
[James’ Reply: It sounds like you are not aware of any way to encourage or preserve the quality of a test process except by formalizing it. It also sounds like you don’t know of any other way to formalize testing other than to “write test cases.”]
I see written test cases as something of check list – so I do not write them too detailed, but I try to mention everything what I have in mind when analysing user story and communicating with the customer. Very often I work on more projects in parallel so I get back to testing once the development is finished and if I had not written down some test cases I would have to remember lots of things and that would be lots of time wasted again.
[James’ Reply: So, then you would be comfortable if I just use a checklist to aid my testing? Is that all you mean by test cases?]
Trust me – I tried both – and at least for me – some amount of test cases written down works like a charm. Every time I tried to test just based on story description or requirement I forgot on something what proves important since I did not remember everything.
[James Reply: The first problem I have with trusting you is that I don’t know what you are talking about. Formal procedures? checklists? I don’t know. The second problem is that I also am a tester with quite a lot of experience and study to inform what I am writing. How can I account for the difference between our two opinions on this matter? Perhaps, instead of trust, we would test each other’s ideas.]
By your logic we could abandon books and start just thinking about everything instead not to get indoctrinated or swayed by someone else’s ideas :-).
[James’ Reply: Clearly you haven’t understood my logic. I will repeat my thesis: “In this article, we explain why creating and performing test cases is not the same thing as doing testing. In fact, test cases are a poor basis for organizing a test process and no basis at all for measuring testing progress. Our industry’s obsession with test cases must come to an end or testing as a field will be never be worthy of respect.”
I am not claiming that test cases are worthless. I’m telling you to stop being obsessed with them. Talk to me about testing, not test cases.]
Hi James, enjoyed reading the article as well as the comments made. I agree that the industry has become “obsessed” with test cases as most see it as a way to measure test progress or “formalize” testing. I think a challenge in trying to change this mindset is that other ways/methods of “formalizing” are not well known and people generally stick to what they know or are comfortable with. Having worked with formal tests cases and still required to work with them, I’m for moving away from them as I feel it stifles exploratory testing especially when we do regression testing. We literally run the same tests with the same steps without bothering to think outside the box.
[James’ Reply: Yes. So, I continue to explain other ways of thinking about and doing testing.
Thank you for helping!]
Hi James,
I found this article (again?) and enjoyed some of your points and the comments as well. I didn’t make it all the way to the end as it’s late and I’m tired but I’m wondering if you’re advocating for no written tests at all (no matter the form) or if you’re just advocating for not obsessing over written tests?
[James’ Reply: I am opposing the obsession.]
When you’re in charge of testing and no one is influencing how you do it…do you write down any tests? If so, how do you use them?
[James’ Reply: Tests cannot be encoded. What can be encoded are notes about testing. I may or may not write test notes or test procedures. It depends on the situation. I document where documentation helps.]
I definitely agree that documented test cases are not the goal and should not be used as a metric. I have a tougher time understanding your point (if your point is) that one should never write down any tests they’ve performed or plan to perform.
[James’ Reply: My point is that if you are a tester, you should perform tests. Performing tests is not the same thing as writing test cases. If you think that writing test cases helps you test, then my question is: in what way does it help? Does it REALLY help?]
I’ve been on teams where we’ve written zero tests and I’ve been on teams where we’ve written hundreds. Can’t say that I remember big advantages to writing zero tests. I definitely don’t like writing complex tests that detail multiple steps just to get to the verification that’s for sure.
[James’ Reply: Okay, here’s the big advantages of not writing anything: you don’t have to maintain it; you don’t have to write it; no one can be misled by it; you are not bound by it; it’s effortless to change. You are constantly not writing things down in your life. In fact, only a small part of what you do is even slightly written down.]
Thanks for the article! It was definitely a good reminder to me as I start my new role that I don’t want to get bogged down in a test case culture.
Daniel
Hi James.
Great article and thanks for taking the time to write it. I can’t agree enough with your conclusions although I want to add a couple of my own.
Test Cases have become, in my experience, a result of poor requirements. It has been easy for Product teams to rely on the Test Team to document how the product should work through test cases as opposed to defining what it should do up front. That said when it comes down to testing everyone wants to pitch it as to how the tester should have tested or could have tested better. Yet no one seems to push back to Product defining better what the product should do.
[James’ Reply: I never understood why anyone thinks that a test case is documentation of how a product works. It may be a demonstration. It may be an example. Documentation is conceptual.]
With a clear set of requirements you don’t need test cases as a pass or fail against the requirement clearly indicates success or failure.
[James’ Reply: I’m not sure what you mean, here, since pass and fail are not something that you get from a requirement. You can only get it from operating the product and evaluating it.]
That all said, I do like to ensure that where ad-hoc testing is undertaken, a high-level summary of what was covered is added to stories in Jira (other tools are available 🙂 ). This just helps to show where testing has covered additional scope outside of the requirements.
[James’ Reply: Nothing about ad hoc testing or exploratory testing implies that you go outside the scope of requirements. If I test outside the scope of requirements I’m not doing my job. I may test outside the scope of the EXPLICIT requirements, but only if I am within the TACIT requirements. Also, nothing about scripted testing implies that it has any different relationship to requirements than exploratory/ad hoc testing does. Testing is never simply a matter of “doing what the requirement says to do” because requirements are not in any way test procedures. They are statements of what we want.
A test is an activity wherein we try to discover the relationship between the product and its tacit and explicit requirements. We look for bugs.
A professional tester should be ready to explain his testing, yes. That may be documented.]
What becomes important is clear details of how to reproduce a bug found along with details of test data and environment config. Therefore the end result is a light touch on manual scripted test for new features, reproducible bugs and automated tests for regression coverage which detail the test cases as part of comments in the script where necessary.
[James’ Reply: I don’t use the term “manual” testing, because it is too ambiguous and also tends to connote unskilled labor. My testing is experiential and interactive. It’s an investigation. I use tools and I use my eyes and hands, too.]
Dan