You also say that the quality engineers “test new server-side features in production”. Why wouldn’t you test this in a non-production environment? What happens if your engineers find critical stop-ship issues? You’ve already shipped.

I’m with James, if you have learned about your customer and your product over time, this is great. But as your product gains more features and interacts with new (an unknown) components, interfaces, applications, etc. what are you able to learn after the fact, but before going to production? This is where testers (or quality engineers) bring their game.

@James - this almost feels like the bulk of “uninteresting” tests have been automated and a perfect example of where exploratory testing could find “interesting” issues. But I’m not sure I have a good understanding how the testers fit in. I also feel that this is an example of where testing is getting “pushed” in general. I know at my company, the feeling is automation is the solution to testing. What I’m reading here reinforces that attitude.

Ken

Information of this practical nature is very interesting and useful. Continuous integration and automated testing are of course very hot topics which makes it even better.

Best regards!

I find Timothy’s comment “things our users wouldn’t think are bugs, and things that we will fix” insulting to the testing community. To me, his statement says that bugs fall into two categories. Things that will be fixed and things that won’t be fixed. Every tester in the world already knows that. So, yes, every bug you and Michael found fell into things that will be or won’t be fixed.

However, I am curious as to how his company determines “value to our customers” especially in light of his statement that his company understands which “things our users wouldn’t think are bugs”. Is it because they are still growing revenue? Is it based on analysis of technical support requests? User-community beta testing? Having that level of understanding indicates a level of marketing and business sophistication I do not normally see in the field. To continue the “Quality is Dead” thread, I could assume that so long as IMVU signs up one new customer they could assume that the bugs in their system aren’t of concern to their customers. Infomercials usually work because of great marketing and salesmanship, not necessarily a quality product. IMVU seems to making the same case for itself. I sense a real life portrayal of “Royal Nonesuch”. I wish IMVU well, but I won’t be their customer because I find their service a novelty, with novelty quality (usefulness).

By the way, I am a big fan of continuous integration (CI) development (and applaud IMVU on their level of sophistication with it). I see CI as allowing a minimum level of quality to be established, via the automated tests, to determine suitability to move a code set forward in the development process. Any code set can be taken and manually tested by a testing department. Candidates passing may be candidates for deploying. To me, CI is akin to automated proof reading, not unlike that in Microsoft Word. It can be very sophisticated and identify where rules have been broken. However, it cannot provide a contextual quality assessment on the product being reviewed. Microsoft Word cannot intelligently determine if what I am writing is of quality or not. It can only determine if I have followed its rules for grammatical structure (and usually spelling). What Microsoft Word allows me to do is focus more time on big picture content, flow, and organization. The same goes for CI, with it, I can focus on bigger picture items.

I very much appreciate the passion evident in your writing, and your addition to the discussion around Continuous Deployment. I completely agree that our product has a very high number of bugs; however, I believe that Continuous Deployment and having a high quality product are actually orthogonal issues. Continuous Deployment can let you develop at a fast pace with minimal regressions, significantly less than having a large team of tests execute regression test cases on each deployment. I’m explicitly not saying that Continuous Deployment gives you high quality features, just that it lets you develop at a rapid pace.

This post seems to hinge on the assumption that we don’t even know that our product is buggy. This is far from the truth. I can’t stress this enough, our company is devoted to making the experience better. The bugs that you and and Michael Bolton found fell into strictly two categories: things our users wouldn’t think are bugs, and things that we will fix.

So how did IMVU create a buggy experience in the first place?

Firstly, my Continuous Deployment post detailed the procedures used for deployment to our production website cluster, not for deploying changes to our downloadable windows client software. While we apply the same principles, the deploy process for the client isn’t nearly as polishe. Our company spent almost all of 2007 with minimal changes to the client, instead scaling up the back-end and incrementally building out Continuous Deployment practices.

Secondly, as detailed in Eric Reis’s talk (http://venturehacks.com/articles/lean-startup), IMVU spent it’s formative years focusing it’s entire efforts on understanding what our users actually wanted to do with the product. Most of the original client software development was done with little or no test coverage, and no professional quality assurance; the company was minimizing capital burn rate while maximizing it’s ability to learn. The company traded off “quality” aka bugs for “quality” aka value to our customers.

Thirdly, I’d like to point out that almost all of the content you saw and experienced was created by our users, for our users. In almost all cases, we prefer user-expression over a software developers or testers sense of “correctness”. We rely on our users and our virtual economy to sort out what works and what doesn’t.

In short, we’re building new techniques to continuously evolve a legacy code base while integrating with a unprecedented amount of 3rd party 3d content; the result is nowhere near perfect but we’re working on it.

I’d also like to make it clear that we have a small but highly valued staff of Quality Engineers who provide the human touch: exploratory testing, missed user stories, sanity checks and input into the creation of automated tests to name just a few things they do. While they don’t stand between our engineers writing code and deploying it, they help smoke test every client release. They also test new server-side features in production prior to users seeing them, since we have a careful roll out system for new feature development that balances manual testing, roll out for load testing purposes and A/B testing.

[James' Reply: I appreciate your willingness to post your thoughts on this.

The principle I am trying to protect is that automated tests-- like automated driving software-- does not substitute for a live driver in the driver's seat. In a car, a sleeping driver might not run off the road right away, but in the long run a crash is inevitable. A crash may happen even if the driver is awake and alert, but a driverless vehicle sparks outrage when it crashes, because that doesn't meet a minimum level of responsibility.]