Muhammad Saad wrote an interesting scenario on LinkedIn. I will represent it here in italics and comment on it…
Imagine yourself on the first day at your job as a tester. You are presented with an application to be tested. It’s an ERP application containing 100s of forms and thousands of reports. You begin your exploratory testing by opening a form that contains around 50 different fields.
You try to enter random data in this form which took around 20 minutes. Then you press submit. Voilà!! An error message is shown which looks like an unhandled exception. You become very happy. You proudly note down the steps and report the bug in your bug management system. Great effort, you feel really confident and energetic. You continue the testing until the day ends and find some more bugs. “Amazing first day”, you thought.
That would indeed be a good first day. The first days– and weeks– of your job as a tester are filled with learning. And a big part of that learning is working through the product under test. To test well you must have a strong mental model of the product. You can get that in lots of ways, but playful, open exploration with the product, a process I call “survey testing” in RST, probably will dominate over all other activities.
Now comes the next day, the developer has fixed the issue and released a new version of the build. You test the same form with the same steps and you found that the bug is fixed. You mark it fixed. Great effort. You have contributed to the quality of the product by identifying that bug and as this bug is fixed, the quality is improved.
Well, a lot of other things are happening in that day. You may not get around to verifying that bug fix. Maybe someone else will. Maybe it can wait. You have huge amount to learn and test. Still, it is plausible that you would verify a fix on the second day.
The more experience I have as a tester, the less interested I am in “contributing to quality.” My interest lies in excellent surveillance of the product, so that no problem can hide from me. I find quality itself to be a distracting idea. But I get it. A lot of people are happy about making the product better. So far, so good in Muhammad’s story.
Now comes the third day, and a developer has again released a newer version. Now you again have to test that form to make sure that no regression issue is found. Same 20 minutes. Now you feel a little bored.
Same 20 minutes? Bored? No and no. It’s not the same 20 minutes. I test a little differently every time I go through. I expand and change the data. I think about tools that might help me generate better data. I probably am talking to people who know the product better than I do (if this is a mature product then surely I am not the first to test it, and if it is not a mature product– how can it have hundreds of forms and reports, already?) It’s only the third day and I have about 200 more days of heavy learning ahead of me for a product as complex as Muhammad is describing.
I’m not bored. I’m engaged with the problem of finding ever cleverer ways of testing. I’m in the midst of learning the functional, operational, and technical details of this technology. People who get bored after such a short time are probably not people who love testing or know very much about it. Testing is not pressing buttons and checking output. Testing is an ongoing application of judgment as you collect evidence and make meaning from it.
If you constantly repeat the same shallow testing process you started with, then you will get bored pretty soon, I guess. But my testing grows deeper as it matures, until it levels out and then gets strategically shallow again as I learn exactly what and where to make the key observations.
Muhammad is probably simplifying this story to make his point clearer. Perhaps when he says “third day” we could take him to mean “day 20” and give him the benefit of the doubt.
Now imagine 1 month from now on, newer versions are constantly releasing, and on every release, you have to test this lengthy form plus 100 other forms like this, just to make sure that no regression is there.
Now you feel angry. You feel tired. You begin to skip steps. You fill around only 50% of the total fields. Your accuracy is not the same, your energy is not the same and definitely, your steps are not the same.
And one day, the client reports the same bug in the same form. You feel pathetic. You feel unconfident now. You think you are not competent enough. Managers are questioning your ability.
I have news for you; this is the story of 90% of the testers out there who don’t use code/tools in their testing.
Maybe Muhammad is projecting his own annoyance with the test process onto other testers. I think he’s right about some of those testers. Indeed, some testers may be angry and bored and tired. But I object strongly that his point holds for people who love to test, and study testing, and make it their career.
Regression testing is driven by regression risk. In the absence of regression risk, there is no need to retest. Regression testing is therefore a much lighter burden than he says it is. Unless the developers are completely out of control and reckless, regression risk is not going to be very high. Old bugs rarely re-open; stable areas of the product do not erupt in spontaneous unrest with bugs bursting out like rebels in a Roman province. It’s almost always more important to do the primary testing which uncovers bugs that have been there all along.
Also, retesting is not a process of repeating exactly what you did the first time. It’s better to introduce a lot of variation into your testing. This helps you find new bugs as well as those old bugs that you may have missed the first time around.
If you feel unconfident in yourself, that’s normal for a new tester. An experienced and trained tester knows that testing is hard, and keeps their expectations of progress to a reasonable level. Managers will question your ability even if you are brimming with ability, because those managers probably aren’t expert testers. They don’t have a clear idea how testing works. They may have some foggy idea that it should all be automated.
Except testing cannot BE automated. What can be automated are certain output checks. I agree with Muhammad that automation is important in most projects, although I generally find it’s done better by a toolsmith working under the advice of a tester, rather than the tester himself. You must always bear in mind what automation can and cannot do, and how experiential and interactive testing (what a lot of people call “manual” testing, but I think that word is insulting and inaccurate) is able to find bugs that automation never will.
Regression issues are the most painful issues. We are humans. And we cannot do the same thing with the same energy, speed, and accuracy every day. This is what machines do. This is what automation is required for, in order to repeat the same steps with the same speed, accuracy, and energy as they were repeated the first time.
Regression bugs are NOT the most painful bugs. The most painful bugs are the ones that hurt the users the most. Nothing about “regression” says that such a bug is especially severe. I want to find every bug that matters.
Muhammad is repeating a well worn myth, which I first attacked in print some 25 years ago in my article Test Automation Snake Oil. He seems to think that machine replicate what humans do. They don’t. He seems to think that automation is inexpensive to create and maintain. In fact, it’s rather expensive, especially relative to how few bugs it ever finds.
I repeat: I believe automation is important to consider in every project; and for some projects it’s absolutely critical. But it’s also quite a lot of work. It’s very easy to create a lot of low value checks that then have to be maintained as the product changes. Muhammad has painted a picture of a project that is severely understaffed, and has apparently gone a long time without a test team at all. Getting the testing ship in order is going to take time and management will have to be patient.
Automation should never be allowed to become an obsession in itself. I have a lot of personal experience obsessing over the tools I write (I started my career as a developer and I enjoy writing code to help me test). I know that I need people around me that help me maintain perspective.
As a professional tester, I work on test strategy, test documentation, performing tests, developing test data libraries, developing test tools, configuring my test platforms, reporting bugs, advocating for testability, learning the product, understanding the users. I’m doing a lot of interesting and challenging things! Automation is a small part of my problem.
Jeff Scott says
Typo perhaps; did you mean “word” instead of “work”?
> (what a lot of people call “manual” testing, but I think that work is insulting and inaccurate)
[James’ Reply: Yes. Fixed. Thanks! Typos are hard for me. I look and I look and miss so many of them. Maybe my eyesite is going.]
Which would be worse for a tester:
Loss of eyesite, or websight?
[James’ Reply: Thank you for this incite.]
Andrew Robins says
It is enough to excite 🙂
Do you mean eyesight? 🙂
[James’ Reply: Ugh. I’m not good at websights!]
Robert Day says
I’ve often thought of software testing as being like the role of a test pilot. A test pilot does not get into a new aeroplane and run through a set of checks as to whether a switch turns on the landing lights and another engages the autopilot (we’ll discount the pre-flight checklist here, because that’s routine for every flight in every aeroplane, not related to the test pilot role). Rather, the test pilot’s objective is to fly the aeroplane and report back on its performance and the limits of that performance – how high it will fly, and how fast; how low, and how slow. If you remember the film “The Right Stuff”, test pilots speak of this process as ‘pushing the edge of the (flight) envelope’.
So in the example you’ve related, the tester’s job on the subsequent days of the project isn’t to repeat the same regression tests and get the same results; it’s to take the outcomes of the first day’s testing and then to go beyond them, to stretch the application’s performance envelope. And that’s how to find the limits of what the application can do and where it is likely to fail.
[James’ Reply: I once taught a class of Air Force pilots and flight instructors. They were going to help test combat flight planning software. I loved those guys. Smart and independent.]
Thank you James for this post. It is really insightful. I have some questions regarding some points you mentioned.
– You highlighted a contrast between a mature product and the fact that it might have hundreds of forms and reports. ERP system for example are quite complex and would always include such configuration. What is your point there?
[James’ Reply: Any system, ERP or otherwise, that has “hundreds of screens” and forms and business rules, etc. is the product of a lot of development work, no? This work happens over a long period of time. Therefore, if the first time you see a product it is already very complex, then either you are not the first tester to work on it, or for some reason the development process has not taken testing seriously up until now. Either way, it’s not your fault and it’s not your problem. You either share the testing burden with other people who must have already figured out a lot about the test strategy and tools, OR you do what you can do alone and don’t blame yourself for being on an understaffed project.]
– I 100% adhere to the idea that manual testing is not the right word to use, from now on I will use the words experimential and interactive testing.
[James’ Reply: The word was “experiential”, not experimental. All testing is experimental. Experiential testing means attempting to duplicate the kind of experience that a particular kind of user would have. You want to see what they would see and feel what they would feel. The opposite of experiential testing is instrumented testing, which means that there is something between you and the direct experience of the product, such as an automation tool, like Selenium. Instrumentation leads to a very different experience of using the product.]
– You mentioned that the regression should be lead by the risk of regression. How do you think that would be best put to practice when you have releases with multiple updated areas where it’s not an easy task to know where the risk is? In addition, there are some risks that are not easy to analyze or predict, like those that are not related to code change. Working on complex big applications might make it not an easy task to have an overall overview. What do you think about that?
[James’ Reply: Risk analysis is not easy, at all. We can never know for sure that we have identified the risks correctly. I recently did a risk analysis for a “smart lock.” Then I went out and used that very smart lock in my own home. To my surprise I found that the calibration process had major flaws. I hadn’t imagined it would be complicated at all to calibrate the lock, until I personally experienced it.
To analyze risk we apply various heuristics. An obvious one is “what, specifically, was changed? What could it have impacted? Avoid testing things that are probably not impacted.” I published a list of these heuristics that can be downloaded from this website. Along with a few articles about risk-based testing. Do a search on my downloads page. I also teach some of this in my classes.]
– how to gain testing maturity and upgrade own skills? How to go from the execution level to the more strategic tasks?
[James’ Reply: Well, you know… I do teach classes on this subject. But if you don’t want to spend money, try downloading some of the many documents I offer, here. Try visiting the Rapid Software Testing channel on YouTube.
Specific idea: search for “testopsy” on YouTube and watch one of my videos about that. A testopsy is an important learning practice.]
Sebastian Stautz says
I totally agree on that.
Another tiny misconception from this Muhammad-like people:
The 20 minutes of the first day are not the same as on the second and third.
Once you know the the actions you need less time to execute them.
The first day is about finding out the exact scenario which triggers the bug. I also imply that he spent time thinking about the random data within this 20 minutes and not before. It also includes learning the product in general, how to use the function.
This does not needs to be done for the regression test as he already know the actions and the data to use.
When I repeat the same scenarios from day one I expect to execute them faster (aside bugs found) on the following days.
Because I already know them. No learning and exploration.
Also many people overlook that you need (at least) the first 20 minutes if you want automate actions and checks.
Somehow they believe that automation requires to first have learned to steps you want to automate.
This just as a side note to all the other points you mention.
[James’ Reply: Another thing about this… It seems to me that repetitive work is often pleasant. It can be comforting. To hear programmer’s talk, you’d think that having to type the same letter twice was like a bee sting. I bet we all have little repetitive rituals in our work. I bet that when certain people complain about the dreary nature of “manual” testing, the root cause is probably that they dislike testing– no matter whether it is repetive or different every time.]
Tim Western says
I feel like I have been railing against the Regression boogie man for years. They pick up the idea that what they tested before, must still work, then assume that they must test exactly the same way and wonder why they find nothing else. The level of thinking in many organizations to lead to a testing practice like that is a headache. Do regressions happen? Absolutely, but they usually have a cause behind why they happen, or happen with frequency, and that is often tied back to the dev practices.
James you are right to call out these things. Thank you.
Andrew Robins says
An approach that I have found useful is to guide regression testing with a Visual Coverage model, that expresses coverage targets for different areas, and different risks in human readable language.
We include information about the value delivered by our existing automated checks in these coverage models. Benefits and limitations are clear, and using human readable language prompts excellent questions from stakeholders.
I like the mix of repeatability and variability that occurs when we use this approach. Once you get stakeholder buy in, it can work really well