Michael Bolton reported on our quick test of IMVU, whose development team brags about having no human-mediated test process before deploying their software to the field.
Some commentors have pointed out that the bugs we found in our twenty minute review weren’t serious– or couldn’t have been– because the IMVU developers feel successful in what they have produced, and apparently, there are satisfied users of the service.
Hearing that, I’m reminded of the Silver Bridge, which fell down suddenly, one day, after forty years of standing up. Up to that day, it must have seemed quite reasonable to claim that the bridge was a high quality bridge, because– look!– it’s still standing! But lack of information is not proof of excellence, it turns out. That’s why we test. Testing doesn’t provide all possible information, but it provides some. Good testing will provide lots of useful information.
I don’t know if the IMVU system is good enough. I do know that IMVU has no basis to claim that their “continuous integration” process, with all their “automated test cases” has anything to do with their success. By exactly the same “not dead yet” argument, they could justify not running any test cases at all. I can’t help but mention that the finance industry used the same logic to defend deregulation and a weak enforcement of the existing laws that allowed Ponzi schemes and credit swap orders to cripple the world economy. Oops, there goes a few trillion dollars– hey maybe we should have been doing better oversight all these years!
It may be that no possible problem that could be revealed by competent testing would be considered a bad problem byIMVU. If that is the case, then the true reason they are successful is that they have chosen to offer a product that doesn’t matter to people who will accept anything they are offered. Of course, they could use ANY set of practices to do that.
Clearly, what they think they’ve done is establish a test process through automation that will probably discover any important problem that could happen before they release. That’s why Michael and I tested it, and we quickly verified what we expected to find: several problems that materially interfered with the claimed functionality of IMVU, and numerous glitches that suggested the presence of more serious problems nearby. Maybe its present users are willing to put up with it, or maybe they are willing to put up with it for now. But that’s not the point.
The point is that IMVU is not doing certain ordinary and obvious things that would reveal problems in their product and they promote that approach to doing business as if it’s an innovation instead of an evasion of responsibility.
The IMVU people can’t know whether there are, in fact, serious problems in their product because they have chosen not to discover them. That they promote this as a good practice (and that manual testing doesn’t scale, which is also bullshit) tells me that they don’t know what testing is for and they don’t know the difference between testing and a set of computerized behaviors called “test cases”.
They are setting themselves up to rediscover what many others have before them– why we test. Their own experiences will be the best teacher. I predict they will have some doozies.