Ask Me About Testing
I find that it’s easier and more fun for me to write when someone asks me an interesting question. If you have a question or a burning issue you’d like me to address in a blog post, respond with a comment on this entry. I will either ignore it, interpolate my reply with your comment, or (if I’m really excited) write a whole blog post on it.
The more interested and thoughtful you seem to be about your question, the more likely I am to respond.
– James
September 3rd, 2006 at 3:36 pm
Hello James
I have been studying a lot of litterature on agile development lately and have realized that I subscribe to a lot of the things that are said. I have read some articles regarding how testers can work successfully in agile projects but surprisingly little is written I think. I would be interested to hear your opinion how testing works with agile development. A specific question is if Exploratory testing is not the natural response to agile development?
Regards Toby
[James' Reply: My experience is that capital "A" Agile people are generally hostile to exploratory testing. They often say that kind of testing is the customer's job. Once in a while they might grudgingly tolerate a testing "specialist" on their team. But in their hearts, the Agilistas I'm complaining about want everything to be automated-- and what can't be automated will be ignored.
Exploratory testing, done by skilled independent testers, would actually be a natural fit for Agile projects who wish to have very high quality. And of course ET fits any truly agile project, since ET is all about agility. ET is an adaptive approach to testing.]
September 3rd, 2006 at 11:21 pm
James — I have few questions to ask .. here is first one …
Would you comment on “Paradigms in Automation” — some thing on the lines of Bret’s “Four schools of Testing”.
Here is what I think about this (my raw thoughts )… your views please …
4 schools of automation
==================
Commercial tool Driven automation
=========================
Use automation to save cost of testing cycle
Who are there in this group?
• IT services and ITeS organizations
• Sales and Marketing folks of these companies
• Project managers and Project Sponsors
a. Primarily GUI based automation
b. Suited to IT services and ITeS organization whose main aim of automation is Minimize regression and testing costs.
c. Home grown automation tools and solutions are not feasible for Business reasons – Their primary goal is to use Software applications to run their primary business.
d. Dependent on commercial off the shelf tools
e. Dependent on Frameworks
f. Parameters
i. Cost reduction of Testing cycles and implementations
ii. ROI Driven
Notions and Perceptions:
1. Automation = Scripting
2. Deploy Automation to reduce or eliminate Manual Testing ( read as Regression testing)
3. Cost of Tool = Investment in Automation
4. Recovery of Investment = Number of times you run the Automated tests. So more often you run the test quicker you recover the investment. Higher the percentage of automation – quicker is the Testing cycle – hence quicker you realize ROI.
5. Automation must be re-usable
6. Users need not learn scripting language
7. common standards and common libraries
8. Highly user friendly
9. Our business users should be able to write automation tests
10. Advanced Keyword driven automation
11. Automation is scripting
12. Automated Testing
13. third generation Test automation tool
14. Automated testing lifecycle ATLM
15. Keyword driven Testing
16. with Automation – your test cycle can finish sooner
Homegrown Automation school
======================
Main Theme
Use automation to Test more often ( daily if possible) and cover more platforms, use it as product knowledge/Test repository
Who are there in this group?
• Product Architects and Test Managers, Test Architects
• Product support groups
• Software Design Engineers and tool smiths
notions and perceptions
a. These organizations use automation tool as “daily build and Test� approach.
b. Their primary goal is make software products – automation here becomes their primary means of encapsulation of their test ideas. These automation solutions have longer shelf life.
c. Microsoft, Yahoo, Google and lots of other companies use – mainly internal tools. Ref to Michael Hunter’s notion of Automation
d. Mainly Non GUI based.
e. Performed by developers and SDET’s – Code is as big and as robust as product code.
f. Good mix of Test driven development or xUNIT testing automation and API Tests and regression and system Tests
Notions and Perceptions
1. Automation must be re-usable
2. Scripts should be modular
3. Details of the script hidden from the user
4. common standards and common libraries
5. Our applications are ahead of technology
6. We use home grown automation tool
7. Automation is software development
8. Framework driven automation
9. table driven automation
10. Keyword driven Testing
11. Using framework – will give more time.
Tool Vendor’s School of Automation
=========================
main theme
I sell software products that help to create automation solutions. For me, automation is a CORE business opportunity
• Sell Automation tool as A software Product
• To some extent -responsible for bringing testers to lime light
• Easier the tool usage – more business
• Do lots of spending in R&D and create tools
• Sell other tools like Load testing and Test management
• Believe that Automation is Scripting and tool knowledge is the most important aspect in automation.
Technologies
Open source School of Automation
=========================
main theme:
Automation is part of product development – are open for others to use for their own purposes.
Promote Open source culture.
Stem from open source software Development movement
Has advantages and disadvantages of any open source software
Automation mainly at Unit and API level – little support for GUI based automation
Technologies
PERL
Ruby and others …
[James' Reply: The concept of a school is essentially that of "paradigm" or "world view". A school sees the field according to a certain model and filter. By definition you cannot authentically belong to two schools at once-- it would be like having two religions at once. Different schools organize facts differently and define words differently. My impression is that you have identified, not schools, but approaches. Another word might be strategies.
My breakdown for approaches would be somewhat different, but yours seems okay. I don't think I understand the last one you mention.]Â
September 4th, 2006 at 1:00 am
Hi James,
Testing is about the context, one technic is the best for one app, but the worse for the nextone, but I keep thinking that it has to be a model or an systematic solution to this.
1. I want you to talk us about Exploratory Testing vs Systematic Software testing.
2. What do you think about had in the same team a test designer and a tester who excecutes that test designs as 2 different guys.
Thanks,
[James' Reply: I don't understand question #2.
As for question #1, I'm not sure what you mean by comparing exploratory with systematic. Exploratory testing done well is systematic. So the comparison is: exploratory testing is an example of systematic testing.
Sometimes people call scripted testing "systematic", as if that's the only thing in the world that is systematic. Actually, scripted testing done badly can be quite unsystematic.
Sometimes people call ponderously documented testing "systematic". I think ponderous is more accurate.]
September 4th, 2006 at 3:58 am
1) I recently read a blog entry over at Cem Kaner’s saying that testers need orders of magnitude improvement in their effectiveness. I agree. I’d like to be one of those testers, someday, somehow. Only…
How the heck can that be achieved?
2) I think all three of us (You, Cem and I) share the opinion that the benefits of
a) the test-related elements of the SWEBOK and
b) most “tester certification”
are “less than claimed”, in the masterful understatement employed by a former director of Xerox PARC. The Emperor’s wardrobe is not as advertised.
What is a tester (who has seen the elephant, who is still trying to get hired) _to do_, given the tension between the operating experiences that lead to these beliefs, and the honest desire to improve practice and tell the truth? Seems like a Herculean task to me. Or maybe my saying that is post-dot-com-shakeout sour grapes.
[James' Reply: I think the answer to both of these questions is pretty much the same. Both Cem and I have released training materials. Consider using those materials. Cem's materials (at http://www.testingeducation.com, but soon to be hosted here at Satisfice) are oriented more to self-paced learning.
My approach to testing is called Rapid Testing because it is oriented around maximizing tester productivity. The exploratory approach to testing is part of Rapid Testing.]
September 4th, 2006 at 8:04 am
James,
4th school (approach as you call) seems to be merging with 2nd approach - ie home grown automation (I would put anything that does not use a *costly* third party automation tool - into this approach. Be it a TDD style (in a(A)gile wolrd) automated tests or those developed using an internal framework - they would here in this)
I would still think that these *nearly* qualify to be schools - with whatever little I know or heard about automation (barring some automation done embedded systems - industrial, process automation etc) - the approaches I have identified can become schools …
1. Commercial Tool Driven
2. Home grown Automation
3. Tool Vendor’s Automation
one pressing arguement that these are schools - each one nearly represent (or appear to represent) one community and I have (at least in my experience) seen one person or group practicing two of these approaches at a time (some times because of business reasons or sometimes purely by choice)
Consider these
A believer of Commercial tool automation will not become a tool vendor himself or will attempt to develop his own home grown tool - for business reasons. The moment he decides to do that - let us say after getting frustrated with commercial tool, one may invest in developing a home grown automation or become tool vendor himself (less likely). When that happens - he has changed his religion hence the approach or school.
A tool vendor less likely to use another commercial tool to automate the tests of his software that he markets. Exception could be here - A tool vendor might use a *home grown* automation to test which he eventualy integrate with his main stream software - this is where at given time a tool vendor is practicing two religions (both as a tool vendor and as someone who uses a home grown automation)
A believer of *home grown* automation, generally does not like commercial tools - he is more than happy to write some additional code and be in full control of Automation. This is true in companies like microsoft (where I worked) - one Test manager there, onece remarked - “we are of the opinion that commercial automation tools can not catch up with us in technolgoies and platforms - If we want to automate few test for a new OS or office application, there can not be any automation tool by a commercial tool vendor”
What information, do I need to make these approaches into full fldged schools - I think these have potential to become one. What are your break down of approaches…
Does my question engage you enough to write a brand new blog post?
[James' Reply: Shrini, I think this is something YOU should blog about. Then ask me to reply to your blog post. I would say that I am in the "Agile Test Automation" school, personally.]Â
September 4th, 2006 at 8:11 am
I have seen you using this capital “A” Agile vs agile ….For the benefit of readers of your blog, can you explain the difference between these two communities ?
my understanding is –
Agile - be agile in some specific way - say by dictating that “all the tests be automated” “there is no role of tester in Agile teams - every one writes code- hence the role of tester and developer are merged in one”…
agile - in contrast goes by literary meaning “swift”, “quick” etc - focuses on act of being agile …
cuts down on all “ceremony” based activities in SDLC…..
[James' Reply: I've blogged about this. See Who Stole Agile? and Defining Agile Methodology.]
September 4th, 2006 at 8:39 am
You’re right, and that’s why I want you to talk about it.
Some people had the impression of make sessions of Exploratory Testing is a waste of time, rattle than make it “systematic” (the way Craig described)
Even as Craig said at his book that Exploratory is a good complement for the structured tests.
Why you give us a brief explanation of what is a good exploratory tester.
[James' Reply: Rick Craig hasn't talked to me about exploratory testing. I'm not sure what he knows about it. I can tell you that exploratory testing, done well, IS structured. That's not what differentiates "traditional" testing and exploratory testing.
I think my article Exploratry Testing Explained, which is on my website, covers this topic pretty well. If you have a specific question after reading that article, let me know.]
September 4th, 2006 at 9:10 am
Hi James,
I have a question about integration testing. Most of the time, I am using network resources from my applications. For instance, I have an application which is transferring files from an FTP site. I would like to be sure that if a problem occurs during FTP transfer process, I do not want it to stop working. So I wrote some tests checking those transfer part. Or I have an application that is collecting objects from a JMS queue. Another one fetching rows from a database. I stuck on the points while testing these features because my tests are only repeatable on my machine. I need a FTP, JMS or DB server for my tests to run successfully so out of my box all these dependent tests fail. I would like to learn what is the best way to do when a network resource is needed during testing in order to make it work everywhere, not just on my machine where the resources are already set up?
Gurkan
[James' Reply: I don't see this as a testing question. It's a technology question.]
September 4th, 2006 at 9:12 am
Hello James,
i must admit that I just start to read your website, so please excuse if this is a FAQ, but here it goes.
I was pressed recently to describe how to plan tests.
Given that the tests have to obey restrictions in time and budget the wish for a plannable future seems understandable. I had a hard time to bring the planning of testing to the point. Do you you have any thoughts abot this which you like to share ?
Thanks a lot for your time and your website
Norbert
[James' Reply: I have written about this on my website. Have you seen the articles in my test methodology section? However about my class notes for Rapid Software Testing?]
September 4th, 2006 at 2:58 pm
Hi James,
As an experienced software developer working with enterprise business software, I find that there are few if any testers. In my current consulting work, developers produce the new functionality, test it themselves, then pass it to the client for acceptance testing. The client ‘testers’ are usually business users with no technical (IT) training or experience, so often do a poor job testing. In some cases, they ask us developers to provide test plans for them to use in testing. I’ve pointed out the lack of testers and proper testing to my manager, who agrees, but getting the client to agree to changes in this area is dificult for a number of reasons.
Your thoughts on these challenges is appreciated. I’m particularly interested in your ideas concerning how we developers can improve the quality and amount of testing we do ourselves. Also, is my experience with the lack of testers the norm or the exception, given the type of software I’m involved with.
[James' Reply: Testers are hired generally because someone is worried that a complex product might not work well enough. It's typical for start-up companies to avoid testers for as long as they can. You'll also see that in companies that don't see software as an important part of their business. Also, I suppose there are people with low quality standards out there, trying to run businesses, too.
I can't honestly say that there is a "norm", when it comes to having testers. It varies so much across the industry.
If you want to make a case to hire a tester, then if I were you, I'd work on these angles:
- Do we want to develop and release product more quickly, yet safely? A tester can help.
- Have we experienced angry customers who keep calling about problems? A tester can help.
- Do we wonder what the status of the product really is, from day to day? A tester can help.Â
The challenge will be to find the right kind of tester. Don't hire one of those process weenies who think testing means complaining about programmers while writing a lot of documents that just gather dust. Hire someone who inquisitive, a good communicator, and cares about your customers. Naturally, I think someone who's up to speed on the stuff written by the Context-Driven testing school people are going to be better candidates.]
Â
September 4th, 2006 at 3:02 pm
One of a tester in my country (India) mailed me with a problem she was facing. Through some mail interactions, I extracted the following information -
1) The product under test is a web application that needs to be tested across the available types of browser.
2) Both manual and automated testing is performed.
3) A part of the manual testing involves UI related cases and cases like ensuring the set colour is reflected on the screen…
4) There is a release every week to the customer.
5) All test cases must be executed and sent as a report to the customer.
Problem statement - Customer has asked for proof of manual testing being carried out. How can proof for manual testing be provided?
This is the information I have in hand. The following analysis is based on the limitation of the above points.
I assume the following are the missing information -
1) Scope of testing.
Is proof of manual testing a part of SLA?
2) Purpose of a weekly release.
3) Density of change of code/complexity of a release to its previous/next week release.
4) The head count of testers working on the project.
5) Cost of the project.
6) Customer’s need for proof of manual testing.
7) How helpful is proof of manual testing to a customer?
9) History of bugs that have been reported by the customer which could have been caught in manual testing from the earlier releases.
10) After which release did the customer express the need of proof of manual testing? What were the bugs reported by the customer after that particular release?
11) The extra cost of providing the proof of testing.
12) What type of the proof is the customer expecting?
13) What are the technologies or products that aid in providing a proof of manual testing?
14) Is proof mandatory for all future releases?
15) Impact of high-medium-low severity bugs on the customer/project/cost of the project.
16) Other information, that I could not think in the one hour I spent for analyzing this problem.
My proposed solutions -
1) Logs of tester actions on the web application or a program that logs Keyboard/Mouse actions.
2) A tool/program that can capture screen shots for every action performed by the application is made to run in the background and for every case.
3) Recorded video file of the manual testing carried out.
4) Manual testing performed on customer site.
5) I came across a tool sometime back, that generates a script on every manual action performed and instead of re testing manually, the stored script can be run or in other words, its like first time you test a product manually and the user actions are converted as script. This can be a good way of providing proof to the customer by sending the scripts with the time stamps.
6) Educating the customer about the extra time/money that may take to provide proof of manual testing and get to know his further willingness about providing a proof of testing.
[James' Reply: I would say that testers don't deal in proof. We deal in evidence-- relevant evidence that supports our clients in making decisions or solving problems. We weave compelling stories around that evidence.
I like your idea of recording the testing with tools. This is easy to do with web applications (try using Burp Proxy, Spector, or TestExplorer), however, I have had virtually no success with script recorders. One little change (in the data, the code, or the state) and the scripts break.]
Now, James it would be great to have your views for the above mentioned problem. It would also be great if you could share your experiences with customers who put up strange requests/orders.
Now for some general questions with a slight bias from the above problem -
1) How would you handle a customer, if he/she does not have any knowledge about testing and demands something out of the scope? In this case, is it a good idea to educate the customer about testing?
2) If a request or demand of a customer is rejected, then is it considered to be a so called *bad way* to deal with the customers?
Many thanks in advance since your reply is going to be a great learning experience. It could either be realization of mistakes or learning of new ways of thinking and doing something.
[James' Reply: I think the only reason I would reject a customer's request is if it is outside the scope of what I am paid to do. For instance, I'm not paid to mislead my clients, nor to do incompetent work. I simply don't sell those services. So if I think a customer is requesting that, I let them know that they should replace me with someone paid a tiny fraction of my rate who will be happy to do bad work for them.
I listen carefully and sympathetically to my clients. I sometimes have to explain testing to them. I sometimes make a counteroffer when I don't think I can do what I'm being asked to do. I also recheck my assumptions.]
September 4th, 2006 at 4:54 pm
I’ve been doing alot of investigating into tools to manage the test process. I often do not work on projects that require the costly and often over complicated and inflexible systems/tools.
I’ve found the best tools out there were not originally designed for software testing, for example, wikis & blogs. Other tools of interest to me are Selenium and Fitnesse. Whilst I am technically capable, I do not class myself as a coder.
I’d be interested to know what tools and/or systems you may use or recommend for a flexible software testing and development project.
Regards,
Rosie Sherry
http://www.drivenqa.com
http://rosiesherry.blogspot.com
[James' Reply: I like the way you think. I've written an article that addresses this. It's called Agile Test Automation.]Â
September 4th, 2006 at 8:38 pm
I’ve always been unclear on how you see exploratory testing fitting into the overall testing process. i.e. do you see it as the primary method of testing an application, or something that is done at key points in the testing process to supplement other testing methodologies? I’ve always been under the impression that you are saying that exploratory testing is sufficient by itself, but I don’t think you’ve actually said that, so I could be way off base.
I like exploratory testing, and think it’s most valuable at specific points in the application lifecycle (e.g. when I first get the app / module), but I rely primarily on automated tests based on structural analysis of the code as well as customer scenarios (including perf, reliability, stress, etc.). I also find that ET is valuable to verify that my other methodologies are sufficient (i.e. if I perform exploratory testing on an area where I’ve written a lot of automation and I’m finding bugs, I know I did a poor job on my automation).
To reiterate my question (because it is likely lost in the rambling above), how do you see ET fitting into the overall testing process? For a bonus question, does your answer change if the software is a one-off release vs. a “shrink-wrap” product that must be maintained for 7-10 years?
[James' Reply: I wonder if we're talking about the same thing when we both speak of "exploratory testing"? To answer your question directly, see, http://www.satisfice.com/articles/where_fits.shtml.
However, it looks from your question that you consider exploratory testing to be a sort of technique that is distinct from other techniques. Exploratory testing is not a technique, but rather an approach. It the polar opposite of scripted testing, but it lies on a continuum with scripted testing, such that almost all testing done by a human is to some extent exploratory.
To test you must learn, design tests, and execute tests. Exploratory testing simply means that these three activities are occuring in parallel to some degree and are supporting each other. The reason I talk about it is because it involves special skills that, like sex, don't come as naturally as some people think they do. But ET is teachable. (Also, like sex, if it bores you, you're probably doing it wrong.)
For some reason, years ago, some idiot got it into his head that firewalling the test design process from the test execution process was somehow more professional than keeping these activities together. Whomever this was, I'll call him Mr. Clueless About Cognition, he misled a lot of people.
If I want to do good testing, then I don't look for opportunities to stop thinking, any more than while I'm driving I'm wondering how I can take more of my attention off of the road. I do, certainly, look for opportunities to carry out my work reliably, and to extend my reach, and to tell the story of testing more effectively (including cost-effectively). Doing so may involve what I call "scripted testing" to some degree.]
September 6th, 2006 at 12:41 am
Functionality Testing: I often hear from the test team members saying “Functional Specification document is like a *Bible* for us to write test cases - and when can we expect it to be completed” - Well, in my mind - functional spec is not the only document but there are other documents to be referred to as well viz., Business Requirements, Technical Specifications, Use case Scenarios, etc, etc.,
So, what kind of reference documents do you think the test team should be adhered to as base reference for writing test cases before proceeding with testing software applications?
(note: i am not referring to adhoc testing)
[James' Reply: I'm not sure what you mean by test cases (there are many different kinds) or why you would distinguish between them and ad hoc testing in terms of documents you might use.
No matter what kind of testing you do, it is part of your job to be aware of the kinds of information you need to do the job well. Documents that contain information about any layer of the technology you are using are certainly worth considering. For instance, if you intend to test a Windows user interface, consider going through the list of key combinations that has special meaning in Windows.
The bigger issue is that there is probably information that resides nowhere in any document. Your job may be to find that information, as well.]
September 6th, 2006 at 3:52 am
In your article “Exploratory Testing Explained v1.3″, Exploratory Testing is “simultaneous learning, test design, and test execution.” Is this still your preferred definition?
Is there ever a place where (or time when) it is not appropriate to use Exploratory Testing?
[James' Reply: Actually, I'm in the process of updating that article. I found that too many people misunderstood what I meant by "simultaneous". So, now, following the first Exploratory Testing Research Summit, Cem Kaner and I have adopted a new, more descriptive definition. My version of it is this:
Exploratory testing is an approach to software testing that emphasizes the personal freedom and responsibility of each tester to continually optimize the value of his work by treating learning, test design and test execution as mutually supportive activities that run in parallel throughout the project.
As a context-driven test methodologist, I can tell you that it is always possible to imagine a context where a particular practice is a poor choice compared to some alternative. As a good context-driven methodologist, I can give you specifics. For instance, exploratory testing is an interactive activity. Therefore, in any situation where interaction is prevented, or is very expensive, it may behoove you to plan your actions in advance. If a certain test platform costs a thousand dollars a minute to use, you might find it more cost effective to prepare and even script your actions to a large degree.
That's one example, now you think of one.]
September 6th, 2006 at 9:40 pm
James — I occasionally work with teams who claim to do exploratory testing, but I am not sure I would call what they do exploratory testing. It IS exploratory testing in that they are given an application with little to no documentation and asked to evaluate it. However, the problems I have with what they are doing is that they have not spent 5 seconds reading, learning, or studying anything about exploratory testing. They say they are doing exploratory testing because it sounds better than what they are really doing, which is running straight down the happy path as delineated by the main menu and the toolbar, having completely disengaged their brains whilst double clicking the desktop icon. No model building, no heuristics, no real exploration of any kind, just “Does File>>Open properly open the file?” Given that I DON’T consider this exploratory testing, and they DO, but that I do see the grey area of it all, where do you (or do you?) draw that line? They are not doing anything overtly wrong…they just are not doing much very right either.
Sincerely,
David
[James' Reply: Would you say that a drunk driver is not actually driving, because the quality of his driving is poor? There can be bad exploratory testing just as there can be bad driving, but we still call it exploratory testing if it meets the requirements of the definition. I am concerned, as you are, about people who give ET a bad name. I do my best to inform my clients and correspondents about how to do it well.]
September 8th, 2006 at 12:25 am
James — here are one more question from my side
When one says “Testing of this application requires domain knowledge” “How can you test this application when you don’t know (or you do not have all required) business domain knowledge” — How do you handle this situation?
To what extent not having required domain knowledge can affect testing and how a skilled tester overcomes such shortcoming (real or perceived).
Here is what I would say “I may or may not have all required business domain knowledge. But I am good learner and can learn these things faster to be effective as quickly as I can. I can think better, critically analyze better, create models, question those basic and assumptions that a so called “domain expert” has taken for granted. All of this essence of good testing”
Shrini
[James' Reply: I like your answer. I would modify it a bit. I would point out that I have a great deal of background knowledge about software and computers which often enables me to find a lot of problems even without special knowledge; and I would say that, as a tester, one of the things I practice doing is getting the most out of domain experts. A testing expert, working with domain experts, can help them be much better testers.
But keep in mind that to test certain features on certain products may actually require a lot of special knowledge. You need to size up each situation and probably make a more specific response. The one time I tried to test a German accounting program (I don't speak German or do accounting) I lasted five minutes before realizing that I was not fit for that duty.]
September 8th, 2006 at 4:41 am
I was reading the thread “should developers do the test first” and started to think tester’s skills. As a test manager / tester I’m constantly facing the problem that I (in my own opionion) do not know enough of the programming languages, about technical architecture etc. I especially face this when I get involved in performance testing. I have indeed worked as a programmer, but it was nearly 10 years ago, and the language or the tools are not used in my current projects. However, I consider myself as a technical person and compared with many of my colleagues (i.e. test managers / testers) I am one of the most technically skilled. But the question is, what is enough and what should we concentrate in? Test managers and testers need to be excellent in project and people management, programming, tools and - testing. But is there a future for not technically skilled test managers or testers (not talking about user testing here)? Would you recommend for example working as a programmer every now and then?
Sincerely
Anna
[James' Reply: I don't know much about the employment dynamics are for testers, these days. I haven't been a hiring test manager since 1998. There are companies, like Microsoft and Google, who are rumored to believe that one must be a programmer to be a good tester. I think that's an ignorant and self-destructive notion, so I hope it isn't true. Testing is not a subset of programming skill. Testing is applied Epistemology. Testers practice a form of practical philosophy when they test. I would much prefer to take a philosopher and teach him computers (I have done this, actually) than to try and retrain a programmer to think like a philosopher.
On the other hand, if you want to maximize your own personal mobility and power in a technical business, being technical helps. My programming skills do come in handy while I test. But there are many kinds of technical skill. You could learn technical writing, or learn about specific technologies. Learn modeling techniques. You could also work on your discrete math, combinatorics, or graph theory skills.
In general, I am always learning and always developing my personal brand name. You can do this, too.]Â
September 8th, 2006 at 10:17 am
James,
Topic One:
What do you think of this bunch of heuristics:
http://en.wikipedia.org/wiki/Unified_Structured_Inventive_Thinking
The wikipedia entry doesn’t do it justice, you have to get the book and pdfs, do the spark plug exercise, relate it to what you already do, etc. But you seemed to already be informed by it anyway.
Could you relate it to your idea of Technical Exploration(from an early blog of yours), and to your use of heuristics in Rapid Testing. Could you highlight any differences in your definition/concept/use of heuristics and Ed’s…or note if they’re the same. Could you relate it to the “Continuous Innovation” process outlined in Artful Making (Austin/Devin).
[James' Reply: There's not much meat in that article in Wikipedia. Nor can I find much online about this. I have, however, several of the Triz books, so I'm aware of the basic ideas. My style is to try and change the world by making ideas available, whereas I guess the Triz and U-SIT people believe in restricting the flow of information so as to make money. Nothing inherently wrong with that, except the ideas don't penetrate as quickly.
Anyway, to answer your question directly, I believe their idea of heuristics is probably the same as mine. A philosophical difference may be that I think it's more important for people to discover and develop their own heuristics than it is to find the right set of super-heuristics that will solve anything.]
Topic Two:
Developers test all the way up to the moment they write code, then the code writes itself. This testing however is not the same as professional software testers and the products they deliver.
[James' Reply: I don't know what this means. I'm trying to relate it to my experience as a programmer, but nothing is clicking. I would not say that I test before I code (except when I am explicitly doing TDD), nor that the code writes itself.]
Developers have a lot to benefit from studying software testing professionals, but what could testers learn from developers? Could the notion of “heuristics” blend the line between developers and testers(”it’s all heuristics, all the way down”). Would there be any benefit to software testing professionals move closer to the moment when the code writes itself? Perhaps not, beyond a certain point it is no longer cost-effective to have anything written down from the tests, they get too small. To have a profession may mean you have defined products that are paid for. But testing still drives everything at that level, below the level of professional software testing.
[James' Reply: It's not all heuristics, it's all mind; all skill and knowledge. Heuristics give a kind of structure to intellectual work.
Of course, there's plenty that testers can learn from programmers, and particularly the specific programmer who wrote the code they happen to be testing. Programmers could learn from testers, too. A lot of the benefit is not from learning, but from working closely together. Personally, I enjoy testing on a programmer's machine just as he completes a build. I like to test with the programmer. It's one of the most productive things I can do, as a tester.]
September 12th, 2006 at 12:53 am
In your opinion is there any difference between the terms Test Case (or Testcase?) and Test Scenario? What should be the granularity of a test case/scenario?
[James' Reply: The meaning of the words test, test case, test procedure, test scenario, test activity, etc. are not so much a matter of opinion as a matter of convention. I have my own convention, but I've seen many others.
For me, a test is a question that I ask of a product. A test case is a specific variation of a test within a series of related tests. A test procedure is a set of instructions that constitute a test. I don't have any fixed definition for test scenario. Test activity is a term I use to refer to some kind of testing that a tester does; a block of testing work.]
September 12th, 2006 at 4:41 am
Question actually provoked by your answer to another topic “The specification is not the test plan”. So what is the test plan? I’ve seen a lot of templates pretending that test plan describe roles, strategies, approaches, techniques, coverage, etc., etc. However if you work within a stable team in a long iterative project, this all becomes copy-paste-that -everyone-knows. I’ve seen in a lot of projects test plan becomes only a holder for test cases. So I wanted to know if and when do you suggest writing detailed test plans and what the mission you see for them?
[James' Reply: The test plan is not a document. The test plan is the set of ideas that guide your testing. Certainly, you can write those ideas down, and then you would have a documented test plan. When I write a test plan down, I write the most concise text I can. Thick test plans are expensive to create and maintain. Thick test plans distract us from testing. Thick test plans are generally the product of fear (e.g. I'm afraid if I don't write lots of text someone will yell at me) and ignorance (e.g. I don't know what to put into a test plan, so I'll blindly follow this detailed template that I got from the Internet).
I included some examples of test plans and test plan parts in the appendices to the RST class (see my website).]
September 18th, 2006 at 6:17 am
James,
Guide me with suitable scenarios that can be thought of while testing Race conditions in an environment where a single account can be operated by many sub account holders in a bulk email marketing business.
Thanks
[James' Reply: This question calls for a technically specific answer, without providing me enough technical specifics to go on. I would need to know more about the risks you are concerned about, as well as the specific features and elements of the product. I've never tested a spam generation system before.]
September 21st, 2006 at 12:07 am
I have roughly framed out few points to write scenarios ( which would provide more light to the topic ) though there are numerous more .
1) Credit management during simultaneous use of credits - where if that account has certain number of credits then that has to be shared and used among the sub account holders
2) Money transaction management - Simultaneous transactions has to be checked .
3) Simulatenous use of any other resource ( kind of mutual exclusion )
4) Priorities assigned to the users - permision based
5) Lock in period for any resource if being used by a user
[James' Reply: Well, this is still far less detail than I need to give a specific answer. But I can tell you one thing, coming up with ideas like these is not very difficult, compared to the problem of actually testing them. This is where test automation might be vital.]Â
September 25th, 2006 at 5:36 am
Do you find test case management tools useful, helpful? If yes, which one, what kind of? By what kind of I mean what feature do you look for in the tool?
In case not using any tool, how do document your testing?
[James' Reply: Since I don't think in terms of test cases, it should not surprise you that I don't use a test case management tool. In the case of Rational's Test Manager, which is one of the worst products of any kind I have had the displeasure of reviewing, the designer insisted to me that he new what test managers wanted in such a tool, although he had not himself ever led a test project. This was years ago, maybe they've fired that designer and hired someone who cares about testing.
Most test case management tools generate bogus metrics based on number of tests that pass and fail. Those metrics are meaningless, at best. They are designed to reward and reinforce the stereotypes about testing held by upper managers.
I don't think about testing the way most people think of it. Obviously, I think my way is better. So I write my own tools to manage my testing. One of them is the scan tool for session-based test management. A few people have caught on to this approach. See the TestExplorer tool, from Sirius Software, which has extensive test documentation features built into it.
I typically use Excel or Notepad for test documentation. I also use tools that generate tables of test data.]Â
October 3rd, 2006 at 7:49 am
While reading about Exploratory Testing and thinking about it - this definition came to my mind.
“Exploratory Testing is applied Psychology for Testers” - helping to think, analyze and model the problem space. A good ET goes beyond testing as seen from world’s view (external View) and dwells deeply into mind of a testers and attempts to find answers for questions like how good tester test and apply mind in solving testing problems”.
This definition is my answer to a common notion (which you have rightly countered in many places) is that “ET is no big deal - testers (professional !!!?) used this much before the term was invented. ET is a consulting Jargon used by few people to make money”.
What is your opinion on this?
Shrini
[James' Reply: I would say that testing-- all testing, not just ET-- is applied Epistemology. It is also, certainly, applied Cognitive Psychology (by definition, all intellectual activities are, since Cognitive Pyschology is the study of how people think).
When people tell me that ET is no big deal because its "already known", I have to struggle through a couple of reactions before I get to one that is useful. My first reaction is shock, because I've been studying and practicing this subject for years. The first useful reaction I generally have to is ask the person to go ahead and explain exploratory testing to me. If they already know it, surely they can teach it to me. Invariably, they are not able to say much about it, and that's when I can start talking about specific heuristics, terminology, dynamics, and related skills.
Another thing I generally say is that, of course, Cem Kaner and I did not invent the phenomenon of exploratory testing. We invented, and continue to invent, ways of doing it better, ways of talking about it, and ways of teaching it.
As far as money goes, I think anyone who actually looks at this field will immediately see that the people making money on testing are large testing companies, and those companies, as I see it, push expensive methods of heavily documented testing because it's profitable, suitable for lower skilled workers, and because most of their customers are prepared to believe that scripted testing is good testing, however unjustified that belief happens to be.]
October 3rd, 2006 at 11:59 am
I love software testing because it is one of the most creative professions.
And creativity is often mentioned in testing articles, but rarely explained.
My guess is that skilled testing is so tightly coupled with creativity that all material regarding how to test more or less deals with creativity.
E.g. your post “How do you stay sharp as a tester?” could have had the title: “how to maintain and enhance your testing creativity ability”.
Creativity is also a fuzzy word, so it could be difficult to understand statements like: a main advantage of exploratory testing is that it enables creativity.
So, what’s your take on creativity in software testing?
Is an environment with humour necessary to foster creative testing?
[James' Reply: I'm not sure what most people mean by the word creativity. Some people seem to use it to mean the ability to produce new and different ideas. Other people use it to talk about self-expression. For still others it connotes a corrosive fog where intelligence turns to mush.
So, I try to avoid the term. I prefer Ed deBono's term "lateral thinking" and other descriptive terms such as critical thinking, systems thinking, abductive inference, etc. That said, all of the literature on creative thinking certainly applies to testing.Â
I suspect humor helps testing. Look at the nature of humor-- a lot of it is the process of making absurd juxtapositions. Just like I do during an input constraint attack.]
October 11th, 2006 at 1:05 am
Hi James,
I have read a lot of articles by you from satisfice website and i have been developing automation scripts for a long time and i somehow developed a passion for these scripts. I feel that automation should be planned as a SDLC cycle with proper planning and devloping to make the scripts robust.since the clients insist on automation scripts , we try to automate all the manual test cases ..but these would be useless for the next release.
I have following doubts:
1.When should we automate?
2.Once we have the manual test cases, how to check for the feasibility of automating the same/what all should be considered for the feasibility study?
3.And above all, What is the best argument to put to the customer to remove their misconception on automation? Because we really know , the time we spend for developing these scripts , maintain for every release(i.e every 3 months) can be spent usefully on effective and efficient manual testing.
[James' Reply: I think I've covered this subject in my article on agile test automation. You'll find it at http://www.satisfice.com/articles/agileauto-paper.pdfÂ
October 13th, 2006 at 5:18 am
I just had a bad experience today and I want to never have that again.
Some time ago, I received an email from a developer which said:
Hi,
Here are the changes I made to the project:
1.
2.
etc.
Please test that ASAP.
The changes were described in detail.
I printed out the email and started verifying each statement. I had some talks with the dev because I did not understood exactly one of the statements.
After I verified it worked correctly when following the statements, I started thinking what could go wrong. I found some issues, related to the changes described in the email.
I continued, and at some point I decided it is good to go.
Today (more than a week later), I found out there was another thing which was modified and broke.
I found this from the dev’s lead.
He told me that even if the dev told me that the only changes were described in the email, he still modified something else. He asked me that next time to test more thoroughly. He also told me not to believe what is written in “change logs”, as some other “minor” changes can be made.
I was mad at me for not catching the issue, as it was an important issue.
I read the email about the changes again and I know why I missed the issue. I misinterpreted it.
The email talked only about the changes the dev “thought” he made. He also did some “minor tweaks” to other parts of the code, but did not mention those in the mail. Also, the email was imperative that the changes have to be tested ASAP. I talked with him at that time and understood there was some kind of pressure from upper levels to get the changes into production.
Now, I did not find a solution to these kind of problems yet, as it has just happened one hour ago.
First, I thought some kind of change management would solve these kind of issues, but then I decided to build rules only for me.
Why?
Because even if someone says: these are the changes, there can be other changes he/she thinks are irrelevant.
Because tiny changes considered “tweaks” will almost always overlooked when writing what was changed.
I just started defining some rules for me so I can minimize the impact of “minor” changes:
1. Don’t test ASAP.
2. If you have to test ASAP buy time.
3. If you have to test ASAP make sure you ask: is this THE ONLY change?
4. Even if someone says IT IS the only change, also test the important parts.
5. If you don’t have time to test the important parts, buy time.
That time I followed rules number 1 and 2, even talked with the developer about the changes, but did not follow rules 3 and 4.
I would have discovered the issue just by hovering the mouse over some part of the product (which I knew is important), but I didn’t.
I concentrated only on the changes described in the email.
I don’t know if the rules I came up will do the job.
Please, if you have other suggestions to improve the way I/we should handle these kind of situations share them with us.
Thank you,
Victor
[James' Reply: Victor, I'm impressed. I love working with people like you. You coach yourself!
Indeed, I have also found that an off-the-cuff email from a programmer is an unreliable guide to what's been changed. We can generalize that: all communication from a programmer is unreliable. This is true simply as a consequence of the even more general rule that all human communication is unreliable.
Just remember, you are a tester, not a sucker. I like your rules 3 and 4. Personally, I do test "ASAP". You can, too. But you need to know how to do it. One thing I do is prepare a test coverage outline and my test data and tools so that I can remember the right things and do the right things under time pressure.
Some people think of test development as creating written test cases. I think of it as preparing for rapidly testing and re-testing my product. This may include automated tests. It definitely includes exploratory testing that adapts to whatever I suspect has changed or been impacted by changes.
It also helps to get information about changes directly from other sources. For GUIs, I have used a tool called Restorator to automatically detect changes in GUI elements by reading the embedded resources. Occasionally I have tracked changes in the source control system as a way to plan testing.
Now the most important thing... While I think you should develop yourself into a tester who can handle anything, the fact is the project manager screwed up. Not the programmer, and not you. If a product is going out very soon after a change is made, then no competent project manager will fail to institute a change control and change review process that is sufficient to assure that each change is approved, reviewed, and tested. (There is an exception to this, of course. If quality doesn't matter much, then don't bother with change control.)]
October 16th, 2006 at 3:12 am
In fact, I also test ASAP.
I just test ASAP by first preparing a plan and deciding what and how to test.
This time, I tested ASAP without enough planning and time to realize some other things can go wrong.
[James' Reply: My point is that you can prepare yourself with the knowledge and tools to test much quicker, yet better, than traditional testing claims that you can. In particular, I'm saying that the anxiety and confusion that often come over testers who are under time pressure need not be a problem for you.]
I know I have a lot to learn in order to become a good tester, so this episode will help me for sure.And I agree with you, the lead or manager of the dev is the one responsible, but I would be happy to stop this kind of issues going into production next time.
The lead was aware of the fact I was not the one to be blamed, he came to me and told me next time not to trust what devs write in emails.
I did not have to defend myself, it was a talk trying to improve what goes into production. I did not tell him “next time better include ALL changes in the email”, because he was away when the email was sent. Also, I did not suggest him to have a change management control tool, because at that time he was upset and could have been irritated by such a suggestion. I will talk with him and his team about this in a few days :).
I wrote this here because materials I found here and at Cem Kaner’s site helped me a lot building my confidence. I was doing some things without having “rules” supporting my way of doing things. Then, after reading some articles I found here there were “rules” supporting my way of doing things.
Thank you,
Victor
[James' Reply: I commend your attitude. Like you, I look at what I can do personally to make things better, without accepting blame for problems other people have created.]
November 8th, 2006 at 1:10 am
James, in context-driven testing if we have to report on testing coverage that would again be context specific. How can we be sure that we cover all the relevant contexts in our testing? Is there any list of contexts that you try to cover in testing?
[James' Reply: There's only one context that matters-- the one that you are in at the moment. If that changes, then you adjust accordingly. It's like parenting. You don't have to figure out how to parent every child, just the ones that belong to you. The context-specific attitude says adapt to your children and then stop adapting. The context-driven attitude, taken to its logical conclusion, is like a child psychologist's approach. Child psychologists need to know how to adapt to any given child (normal and strange) who walks in the door.
For the same reason, if you figure out how to report coverage on your project in a way that works for you, you can't assume that the same method will work for me, nor do you need to worry about whether it would work for me. You can't tell me "James, I have discovered the right way and you should do it my way, too." What you say, intead, is "James, would you like me to describe some experiences I've had with coverage reporting? I feel good about how I do it, over here in my project."]
November 10th, 2006 at 10:11 pm
good day sir. what are the characteristics of traditional testing and creative testing in relation to personality?
thanks for ur humble reply
[James' Reply: I would need you to explain what you mean by traditional testing and creative testing. I would also need you to describe what you mean by personality.
In my view, the tradition in testing to is treat the process as a form of secretarial work. I see testing as a matter of systematic skepticism and critical thinking. The personality of a consistently successful tester of this kind must be one of being excited (rather than frightened) by complex problems and ambiguous situations.]Â
November 21st, 2006 at 2:46 am
James, I have a small request or if you don’t mind a suggestion. I believe you must have your favorite sites which you would like other testers to read. How about keeping it on your blog? Moreover I hope other testers would also suggest some sites which you would like to share. Here is the one to start with on Systems Thinking: http://wiki.systemsthinking.net/
[James' Reply: I actually don't have a list of favorite sites! I just go to Google when I want something. I have to think more about this.]Â
April 7th, 2008 at 5:57 am
I don’t know whether it is right thing to ask this here or not, I was thinking and googling a lot on this. I want to know that what kind of tools and/or scripting languages Apple might have used or is using to test products like iPhone software which has multitouch capability - no automation tool can do this…Any idea or thoughts.
[James' Reply: Good question. I don't know. If I were them I would have built the software in an inherently testable fashion, so that they weren't trying to automate through the GUI at all. But I'm not them.]
April 9th, 2008 at 6:24 am
Hi James,
Could you please explain in brief (one or two lines) what is ‘inherently testable’.
Thanks.
[James' Reply: "Inherently" is a bit redundant. I could have just said "testable". What I mean, simply, is that the product is designed with testing in mind. This primarily means controllability and visibility features. For example: log files (visibility of internal states) and a scriptable interface (controllability).]
January 26th, 2011 at 1:20 pm
James,
I took the initiative of Team Lead in a Software Testing Project and though it was new,it went fine.
Now,during all these times- I suggested a bunch of process changes,new approaches towards testing,etc.
So please help me with a Template where i can keep all the suggested recommendations,new process methods given by me,etc.IN brief,Recommendation Document Template.
Rohit
[James' Reply: You don't need a template.]