Noteworthy Testing Companies
Now for a word about testing companies.
I’ve worked at two testing companies, not counting my own. I’ve toured several others, and I know which ones have a standing among the people who themselves have a public reputation in the testing field. Occasionally I am approached for an endorsement. I’m happy to give endorsements, but only if I have good evidence about the quality of their work.
Here’s what I can say right now: Most outsource testing companies claim to be skilled at testing. Yet, few of them appear to have any basis for that claim. For the most part, testing companies seem to be sweatshops of low paid keyboard pounders and script runners. However, there are some interesting exceptions.
“Low Cost” Testing Companies
I poll test managers about their experiences with outsourcing to “low cost” testing companies, typically in India. I’ve seen a remarkable pattern: NOBODY is satisfied. I mean literally nobody; not a single test manager that I have spoken with about this (on the order of 150 managers). Instead, I generally hear a drawn out sigh accompanied by sad head shakes of quiet desperation while the poor manager tells me how outsourcing has created a lot of paperwork and produced little benefit. I hear things like “Those people need their tests fed to them through a straw” and “They say ‘yes’ even when they don’t understand what we are asking for.”
That changed this week. I was teaching in Houston, a couple days ago, and the subject turned to offshore testing. No less than three test managers began gushing about the excellent exploratory testing they were getting from their testing service in Bangalore. (Well, they called it “ad hoc testing”, but I can forgive that).
After some prodding, I got them to tell me the name of this company. Drum roll! It’s the Bangalore office of MphasiS.
I cross-examined the three managers. Each of them had specific incidents and examples to share. I suppose I must stop telling people that I’ve never heard or seen offshore testing work well.
Congratulations, MphasiS. As for the rest of you, keep trying. (Hint: Keep your brains turned on to full blast while you’re testing. Trust me, your American clients will praise you, too.)
I have visited Mindtree and Wipro in India, and seen some encouraging signs. But I haven’t yet heard any customer testimonials about them.
“High Skill” Testing Companies
So far there are only two firms that I can fully vouch for:
- Logigear (run by Hung Nguyen)
- Quardev (where my brother, Jonathan, runs their rapid testing service).
Both labs are based in the United States, and are run by people who write and speak in the field. I believe both of them work with offshore staff, too. Logigear has an office in Vietnam.
I have also worked with Sopra Newell & Budge and Transition Consulting Limited, both in the U.K. I have not seen enough of their work to endorse them outright, but there are particular impressive people working there who give me hope: Graham Freeburn at SN&B; Mark Garnett and Stewart Noakes at TCL.
(My company also does software testing, but I have no standing staff. I specialize in testing for court cases and other high stakes situations, so I put together a team as needed for each project, or else borrow staff from Quardev.)
July 17th, 2006 at 7:09 am
I work for a company called readytestgo. We do both performance and functional testing. All of our customers are really very satisfied and referable.
So please validate any comment before publish anything in the blogger.
[James’ Reply: How should I validate your own statement? When you say “all” of your customers are satisfied, may I assume that you will stop saying that as soon as one single customer of yours is not satisfied? Or should I assume you have no customers, or just one or two? Or perhaps all your customers are relatives of yours?
I consider myself an excellent tester. You’ll hear no false modesty from me. Yet even I cannot say that ALL my customers would describe themselves as satisfied. I would only claim, without proof, that all of them would have been satisfied if they shared my sense of what should be satisfying to them.
It’s not particularly difficult, anyway, to fool a customer into being satisfied, if they don’t know what to expect. That’s why I can’t fully endorse a company unless I’ve worked with them, and why even a customer testimonial must be cross-examined, first.]
July 17th, 2006 at 11:43 am
I was amazed reading what most managers feel about “low cost” testing companies (i.e. offshore), as someone who’s been working with offshore QA teams constantly I feel the same. I do however like to see more (or any) papers indicating hard facts and numbers analyzing the profitability/benefits of off shoring QA projects. You also didn’t mention anything about what type of testing projects they outsourced or what practices they used to overcome typical offshore project problems (communication for example) which may jeopardize the entire project, but it seems out of the scope of your blog’s title.
I feel that this phenomenon of off shoring software/testing projects is over hyped and in the long run the benefits (if any) are extremely low. I just wish there was a valid paper to indicate this way or the other.
[James’ Reply: I would love to do such a study, but I doubt you’ll ever see one published. No one responsible for bad decisions on a large scale wants to pay a consultant to verify and detail the badness of those decisions. And certainly they don’t want it made public! Funding for critical case studies usually comes from an auditing or investigating authority following a disaster of some kind, or as part of some legal proceeding. Results from such a study are generally confidential, unless it’s government work.
I have actually done critical studies of testing practices, and not one am I allowed to share with you.]
Finally you come up with a good indicator that performing exploratory testing by low cost companies is successful. I don’t know what to make of it, at first I thought exploratory testing requires highly qualified testers who are familiar with this technique. On the other hand you imply that “low cost” offshore teams lack this ability (I think that your google’s TechTalk and the comment you made about the tribe they found in South America emphasis this).
So is there a contradiction here?
I.H.
[James’ Reply: I have no way of knowing whether MphasiS is actually skilled at ET. But ET is the kind of thing that you can produce wonderful results in without having any special skill. Skill in ET allows you to produce consistently excellent results, regardless of what you’re testing, and allows you to link ET with project management, teach others to do it better, etc. The first leap of ET is to allow yourself to admit you are doing it, and to do it in the open. MphasiS seems to be at that first stage, at least. Maybe they have taken it further.
I bet that it isn’t the whole company, but rather just a handful of people who are making it work.]
July 17th, 2006 at 2:22 pm
Hi James,
As a tester from India, I would want to add my views on the topic of - testing services outsourced to Indian companies that sell its testers sweat for a cheap rate.
You said you did poll the managers and they said “Those people need their tests fed to them through a straw” and “They say ‘yes’ even when they don’t understand what we are asking for.” - good to see most(all) of them speaking from heart.
At times, I am disappointed at the current state of Indian Testing community and unfortunately, I am aware, the problem is population. ( this info about mPhasis is a silver lining)
There are lakhs of engineers pumped out of every year and testing is the entry point to the industry to most of them who are looking for jobs roaming on roads. Some change their profile to development but some of them to make the industry unfortunate, are unable to make a shift to development ( not all from India but to be honest, most of them). When a majority of these people get experience, Indian software industry is so promising, at least as of now that they have a job somewhere for a better pay, as we are payed more than a month of salary to refer someone who is an experienced tester.
The concept of spoon feeding started in schools for us and we wanted it to continue in engineering degree colleges, we were happy, it did continue. Today we are sad to know and realize that we continue it in the industry too.
I am happy that, I have helped around 200 testers in India to come out of the mindset of spoon feeding and very proud to see them thinking on their own and asking questions before they say “Yes”.
Someday we would be giving you the best quality and satisfaction, yet for a cheaper price as we would be more passionate to work rather than associating more profit to work.
July 20th, 2006 at 3:06 pm
Hi James,
I appreciate your feedback about the testing industry in my country (India). It is valuable feedback like this that will put India on the right course to being a globally competitive outsourcing location for testing also.
However I do want to analyse the current state of affairs in the light of the evolution of an outsourcing industry. Unlike the other domains where India is fast becoming the choice for outsourcing like software development outsourcing and offshoring, BPOs & KPOs) testing is still in its nascent stages in India. It lacks the learning curves that the other domains have undergone over the decade. Outsourcing is a beautiful business concept where countries mutually benefit. However it requires that people who provide these services develop these new skillsets “On the job”. Though our universities train people for these skillsets the global divide that exists between the locations of expertise (USA for example) and the local universities results in theoretical knowledge being imparted than useful practical knowledge. (Especially during the early days)
As a result “On job” learning is higher. Unfortunately this means that more mistakes are commited by people during the job until they acquire the necessary expertise. This is like the chicken and egg problem. People wont learn the skillset unless they commit mistakes and clients dont like mistakes.
The current state of testing in India is a result of these issues. Also testing is a role which like any other advanced skillset has multiple dimensions to it. This increases the time to expertise. Added to these are the cross-cultural issues.
Exploration and research were not a common phenomenon in India. There is always the initial hesitancy when posed with unknown situations.
But, looking at the progress we’ve made in the software development and other fields i can assure you that India will one day have a globally competitive brand in test outsourcing too.
I remember, during the initials days of software outsourcing (95-97s) we used to get very similar feedback on our coding standards too. The number of bugs per lines of code written in India made headlines those days. But we’ve come a long way from there. Our industries are today being lauded for the quality standards that we are achieving.
Im sure with consistent feedback and space for improvement these issues can be resolved. Please keep the comments comming.
Your ardent fan,
Nandan.
[James’ Reply: Thanks, Nandan. That’s helpful information. I do believe India is becoming more competitive. I understand they are also becoming generally more expensive. Indians will increasingly need to compete on quality as well as cost in order to fend off challenges from other, less expensive countries.
I’m trying to set up a lecture tour of India, together with Cem Kaner and Michael Bolton. We’re still assessing the demand for this and the sponsorship and the logistics. But my fantasy is to have the opportunity to speak to at least a few big crowds of Indian testers, to directly convey to them my own passion for an intellectual approach to software testing– one which I believe Indians would be well able to pursue, if and when they apply themselves to it.]
July 24th, 2006 at 2:41 pm
Thinking about opportunities of outsourcing to “Low Cost” company one should always keep in mind that in most cases he will face “cheap and dirty” service. On the other hand I strongly disagree with colleagues who believe that “in the long run the benefits [of offshoring] are extremely low”. The successl is not just a matter of finding reliable and skilled offshore partner. It is also a matter of how customer company is prepared for effective collaboration. Some companies are thinking that outsourcing will work automatically right away after they invest $5K, $10k, $100k, etc. to hire 100, 500, 1000, etc. guys in India or elsewhere. They get burned and become disheartened about outsourcing after that. But they should stop, think and carefully elaborate their approach, wheather they need to change somthing in themselves to get ready for outsourcing?
For example, from our experience proper communication is always a challenge whenever outsourcing is considered. Introduction of outsourcing might request changes in organization structure, communication, thinking and general way of getting things done. Unlike in-house work outsourcing requires more careful communication, all aspects should be explained in details, re-asked or even re-iterated to exclude misunderstanding and ensure the common vision. The knowledge transfer being done internally takes much less effort then externally (e.g. the things previously existed just as corporate culture now should be put on paper!). But this effort is worth paying since the things agreed in details once are reused and adopted further more and more. A short time after off-shore team adopts enough knowledge of the company to work as efficiently as your own department or even better.
As for the case when “Those people need their tests fed to them through a straw” and “They say ‘yes’ even when they don’t understand what we are asking for.”. Our US customers said the almost the same words talking about previous experience with outsourcing. What I can recommend is to work harder on selecting the right partner, because nothing will work untill you have really bright guys on the other side. Talk to them, visit them, meet authorities and regular workers, give them pilot projects, gather recommendations from the 3d parties in order to estimate what level of skills and devotion they can contribute. Only if you find a good team with bright minds and strong organization you can see the light side of the Moon: drastically improve your overall performance and quality with low cost.
My best regards!
March 10th, 2008 at 2:05 pm
I am just starting a testing comapny due to my experience in manual testing and good investment capacity
Is it possible to contact you for for some step by step help in the starting period and how much do you charge for consultancy?
[James’ Reply: My rates vary. I charge differently depending on whether I have to travel, how far I have to travel, and how interesting the work is. I charge differently for classes or interactive consulting. Contact me at james@satisfice.com, and tell me a little about your situation: where you are located and what kind of thing you’re interested in having me do. Thanks!]