Muhammad Saad wrote an interesting scenario on LinkedIn. I will represent it here in italics and comment on it…
Imagine yourself on the first day at your job as a tester. You are presented with an application to be tested. It’s an ERP application containing 100s of forms and thousands of reports. You begin your exploratory testing by opening a form that contains around 50 different fields.
You try to enter random data in this form which took around 20 minutes. Then you press submit. Voilà!! An error message is shown which looks like an unhandled exception. You become very happy. You proudly note down the steps and report the bug in your bug management system. Great effort, you feel really confident and energetic. You continue the testing until the day ends and find some more bugs. “Amazing first day”, you thought.
That would indeed be a good first day. The first days– and weeks– of your job as a tester are filled with learning. And a big part of that learning is working through the product under test. To test well you must have a strong mental model of the product. You can get that in lots of ways, but playful, open exploration with the product, a process I call “survey testing” in RST, probably will dominate over all other activities.
Now comes the next day, the developer has fixed the issue and released a new version of the build. You test the same form with the same steps and you found that the bug is fixed. You mark it fixed. Great effort. You have contributed to the quality of the product by identifying that bug and as this bug is fixed, the quality is improved.
Well, a lot of other things are happening in that day. You may not get around to verifying that bug fix. Maybe someone else will. Maybe it can wait. You have huge amount to learn and test. Still, it is plausible that you would verify a fix on the second day.
The more experience I have as a tester, the less interested I am in “contributing to quality.” My interest lies in excellent surveillance of the product, so that no problem can hide from me. I find quality itself to be a distracting idea. But I get it. A lot of people are happy about making the product better. So far, so good in Muhammad’s story.
Now comes the third day, and a developer has again released a newer version. Now you again have to test that form to make sure that no regression issue is found. Same 20 minutes. Now you feel a little bored.
Same 20 minutes? Bored? No and no. It’s not the same 20 minutes. I test a little differently every time I go through. I expand and change the data. I think about tools that might help me generate better data. I probably am talking to people who know the product better than I do (if this is a mature product then surely I am not the first to test it, and if it is not a mature product– how can it have hundreds of forms and reports, already?) It’s only the third day and I have about 200 more days of heavy learning ahead of me for a product as complex as Muhammad is describing.
I’m not bored. I’m engaged with the problem of finding ever cleverer ways of testing. I’m in the midst of learning the functional, operational, and technical details of this technology. People who get bored after such a short time are probably not people who love testing or know very much about it. Testing is not pressing buttons and checking output. Testing is an ongoing application of judgment as you collect evidence and make meaning from it.
If you constantly repeat the same shallow testing process you started with, then you will get bored pretty soon, I guess. But my testing grows deeper as it matures, until it levels out and then gets strategically shallow again as I learn exactly what and where to make the key observations.
Muhammad is probably simplifying this story to make his point clearer. Perhaps when he says “third day” we could take him to mean “day 20” and give him the benefit of the doubt.
Now imagine 1 month from now on, newer versions are constantly releasing, and on every release, you have to test this lengthy form plus 100 other forms like this, just to make sure that no regression is there.
Now you feel angry. You feel tired. You begin to skip steps. You fill around only 50% of the total fields. Your accuracy is not the same, your energy is not the same and definitely, your steps are not the same.
And one day, the client reports the same bug in the same form. You feel pathetic. You feel unconfident now. You think you are not competent enough. Managers are questioning your ability.
I have news for you; this is the story of 90% of the testers out there who don’t use code/tools in their testing.
Maybe Muhammad is projecting his own annoyance with the test process onto other testers. I think he’s right about some of those testers. Indeed, some testers may be angry and bored and tired. But I object strongly that his point holds for people who love to test, and study testing, and make it their career.
Regression testing is driven by regression risk. In the absence of regression risk, there is no need to retest. Regression testing is therefore a much lighter burden than he says it is. Unless the developers are completely out of control and reckless, regression risk is not going to be very high. Old bugs rarely re-open; stable areas of the product do not erupt in spontaneous unrest with bugs bursting out like rebels in a Roman province. It’s almost always more important to do the primary testing which uncovers bugs that have been there all along.
Also, retesting is not a process of repeating exactly what you did the first time. It’s better to introduce a lot of variation into your testing. This helps you find new bugs as well as those old bugs that you may have missed the first time around.
If you feel unconfident in yourself, that’s normal for a new tester. An experienced and trained tester knows that testing is hard, and keeps their expectations of progress to a reasonable level. Managers will question your ability even if you are brimming with ability, because those managers probably aren’t expert testers. They don’t have a clear idea how testing works. They may have some foggy idea that it should all be automated.
Except testing cannot BE automated. What can be automated are certain output checks. I agree with Muhammad that automation is important in most projects, although I generally find it’s done better by a toolsmith working under the advice of a tester, rather than the tester himself. You must always bear in mind what automation can and cannot do, and how experiential and interactive testing (what a lot of people call “manual” testing, but I think that word is insulting and inaccurate) is able to find bugs that automation never will.
Regression issues are the most painful issues. We are humans. And we cannot do the same thing with the same energy, speed, and accuracy every day. This is what machines do. This is what automation is required for, in order to repeat the same steps with the same speed, accuracy, and energy as they were repeated the first time.
Regression bugs are NOT the most painful bugs. The most painful bugs are the ones that hurt the users the most. Nothing about “regression” says that such a bug is especially severe. I want to find every bug that matters.
Muhammad is repeating a well worn myth, which I first attacked in print some 25 years ago in my article Test Automation Snake Oil. He seems to think that machine replicate what humans do. They don’t. He seems to think that automation is inexpensive to create and maintain. In fact, it’s rather expensive, especially relative to how few bugs it ever finds.
I repeat: I believe automation is important to consider in every project; and for some projects it’s absolutely critical. But it’s also quite a lot of work. It’s very easy to create a lot of low value checks that then have to be maintained as the product changes. Muhammad has painted a picture of a project that is severely understaffed, and has apparently gone a long time without a test team at all. Getting the testing sorted out is going to take time and management will have to be patient.
Automation should never be allowed to become an obsession in itself. I have a lot of personal experience obsessing over the tools I write (I started my career as a developer and I enjoy writing code to help me test). I know that I need people around me that help me maintain perspective.
As a professional tester, I work on test strategy, test documentation, performing tests, developing test data libraries, developing test tools, configuring my test platforms, reporting bugs, advocating for testability, learning the product, understanding the users. I’m doing a lot of interesting and challenging things! Automation is a small part of my problem.