Michael Bolton reported on our quick test of IMVU, whose development team brags about having no human-mediated test process before deploying their software to the field.
Some commentors have pointed out that the bugs we found in our twenty minute review weren’t serious– or couldn’t have been– because the IMVU developers feel successful in what they have produced, and apparently, there are satisfied users of the service.
Hearing that, I’m reminded of the Silver Bridge, which fell down suddenly, one day, after forty years of standing up. Up to that day, it must have seemed quite reasonable to claim that the bridge was a high quality bridge, because– look!– it’s still standing! But lack of information is not proof of excellence, it turns out. That’s why we test. Testing doesn’t provide all possible information, but it provides some. Good testing will provide lots of useful information.
I don’t know if the IMVU system is good enough. I do know that IMVU has no basis to claim that their “continuous integration” process, with all their “automated test cases” has anything to do with their success. By exactly the same “not dead yet” argument, they could justify not running any test cases at all. I can’t help but mention that the finance industry used the same logic to defend deregulation and a weak enforcement of the existing laws that allowed Ponzi schemes and credit swap orders to cripple the world economy. Oops, there goes a few trillion dollars– hey maybe we should have been doing better oversight all these years!
It may be that no possible problem that could be revealed by competent testing would be considered a bad problem byIMVU. If that is the case, then the true reason they are successful is that they have chosen to offer a product that doesn’t matter to people who will accept anything they are offered. Of course, they could use ANY set of practices to do that.
Clearly, what they think they’ve done is establish a test process through automation that will probably discover any important problem that could happen before they release. That’s why Michael and I tested it, and we quickly verified what we expected to find: several problems that materially interfered with the claimed functionality of IMVU, and numerous glitches that suggested the presence of more serious problems nearby. Maybe its present users are willing to put up with it, or maybe they are willing to put up with it for now. But that’s not the point.
The point is that IMVU is not doing certain ordinary and obvious things that would reveal problems in their product and they promote that approach to doing business as if it’s an innovation instead of an evasion of responsibility.
The IMVU people can’t know whether there are, in fact, serious problems in their product because they have chosen not to discover them. That they promote this as a good practice (and that manual testing doesn’t scale, which is also bullshit) tells me that they don’t know what testing is for and they don’t know the difference between testing and a set of computerized behaviors called “test cases”.
They are setting themselves up to rediscover what many others have before them– why we test. Their own experiences will be the best teacher. I predict they will have some doozies.
James,
I very much appreciate the passion evident in your writing, and your addition to the discussion around Continuous Deployment. I completely agree that our product has a very high number of bugs; however, I believe that Continuous Deployment and having a high quality product are actually orthogonal issues. Continuous Deployment can let you develop at a fast pace with minimal regressions, significantly less than having a large team of tests execute regression test cases on each deployment. I’m explicitly not saying that Continuous Deployment gives you high quality features, just that it lets you develop at a rapid pace.
This post seems to hinge on the assumption that we don’t even know that our product is buggy. This is far from the truth. I can’t stress this enough, our company is devoted to making the experience better. The bugs that you and and Michael Bolton found fell into strictly two categories: things our users wouldn’t think are bugs, and things that we will fix.
So how did IMVU create a buggy experience in the first place?
Firstly, my Continuous Deployment post detailed the procedures used for deployment to our production website cluster, not for deploying changes to our downloadable windows client software. While we apply the same principles, the deploy process for the client isn’t nearly as polishe. Our company spent almost all of 2007 with minimal changes to the client, instead scaling up the back-end and incrementally building out Continuous Deployment practices.
Secondly, as detailed in Eric Reis’s talk (http://venturehacks.com/articles/lean-startup), IMVU spent it’s formative years focusing it’s entire efforts on understanding what our users actually wanted to do with the product. Most of the original client software development was done with little or no test coverage, and no professional quality assurance; the company was minimizing capital burn rate while maximizing it’s ability to learn. The company traded off “quality” aka bugs for “quality” aka value to our customers.
Thirdly, I’d like to point out that almost all of the content you saw and experienced was created by our users, for our users. In almost all cases, we prefer user-expression over a software developers or testers sense of “correctness”. We rely on our users and our virtual economy to sort out what works and what doesn’t.
In short, we’re building new techniques to continuously evolve a legacy code base while integrating with a unprecedented amount of 3rd party 3d content; the result is nowhere near perfect but we’re working on it.
I’d also like to make it clear that we have a small but highly valued staff of Quality Engineers who provide the human touch: exploratory testing, missed user stories, sanity checks and input into the creation of automated tests to name just a few things they do. While they don’t stand between our engineers writing code and deploying it, they help smoke test every client release. They also test new server-side features in production prior to users seeing them, since we have a careful roll out system for new feature development that balances manual testing, roll out for load testing purposes and A/B testing.
[James’ Reply: I appreciate your willingness to post your thoughts on this.
I’m happy to hear that you have testers. I’m confused that you would use the phrase “stand in the way.” Testers do not stand in anyone’s way on any project. Testers do not run projects. Testers provide information. So, what I’m wondering is, when you release a product, what do you know about it? The purpose of testing is to make informed, responsible decisions about products possible. It sounds like you release the product without knowing much about it, other than that it passed some automated tests.
The answer may be that you do know enough about the product, because you accumulate knowledge about it over time, and your change control protocol is very strict so as to minimize the spoilage of that knowledge. Am I right? If so, I can respect that, as long as you are mindful of the risk.
The principle I am trying to protect is that automated tests– like automated driving software– does not substitute for a live driver in the driver’s seat. In a car, a sleeping driver might not run off the road right away, but in the long run a crash is inevitable. A crash may happen even if the driver is awake and alert, but a driverless vehicle sparks outrage when it crashes, because that doesn’t meet a minimum level of responsibility.]
First, I don’t normally comment on anyone’s grammar, but I find it humorous that in Michael’s blog he does refer to you as Jon in one sentence. The rest of my comments are regarding Timothy’s response.
[James’ Reply: He’s referring to my brother. I appreciate the apologetic note you wrote when you discovered that, but no worries. I’m proud to be mistaken for my brother, once in a while.]
I find Timothy’s comment “things our users wouldn’t think are bugs, and things that we will fix” insulting to the testing community. To me, his statement says that bugs fall into two categories. Things that will be fixed and things that won’t be fixed. Every tester in the world already knows that. So, yes, every bug you and Michael found fell into things that will be or won’t be fixed.
However, I am curious as to how his company determines “value to our customers” especially in light of his statement that his company understands which “things our users wouldn’t think are bugs”. Is it because they are still growing revenue? Is it based on analysis of technical support requests? User-community beta testing? Having that level of understanding indicates a level of marketing and business sophistication I do not normally see in the field. To continue the “Quality is Dead” thread, I could assume that so long as IMVU signs up one new customer they could assume that the bugs in their system aren’t of concern to their customers. Infomercials usually work because of great marketing and salesmanship, not necessarily a quality product. IMVU seems to making the same case for itself. I sense a real life portrayal of “Royal Nonesuch”. I wish IMVU well, but I won’t be their customer because I find their service a novelty, with novelty quality (usefulness).
By the way, I am a big fan of continuous integration (CI) development (and applaud IMVU on their level of sophistication with it). I see CI as allowing a minimum level of quality to be established, via the automated tests, to determine suitability to move a code set forward in the development process. Any code set can be taken and manually tested by a testing department. Candidates passing may be candidates for deploying. To me, CI is akin to automated proof reading, not unlike that in Microsoft Word. It can be very sophisticated and identify where rules have been broken. However, it cannot provide a contextual quality assessment on the product being reviewed. Microsoft Word cannot intelligently determine if what I am writing is of quality or not. It can only determine if I have followed its rules for grammatical structure (and usually spelling). What Microsoft Word allows me to do is focus more time on big picture content, flow, and organization. The same goes for CI, with it, I can focus on bigger picture items.
Thank you James and Michael for writing your blog posts and thank you Timothy for answering.
Information of this practical nature is very interesting and useful. Continuous integration and automated testing are of course very hot topics which makes it even better.
Best regards!
@Timothy – You say that the quality engineers “smoke test” the product. What does this mean? In my world, the smoke test is one of the first things automated. Why wouldn’t you include that in your existing automation?
You also say that the quality engineers “test new server-side features in production”. Why wouldn’t you test this in a non-production environment? What happens if your engineers find critical stop-ship issues? You’ve already shipped.
I’m with James, if you have learned about your customer and your product over time, this is great. But as your product gains more features and interacts with new (an unknown) components, interfaces, applications, etc. what are you able to learn after the fact, but before going to production? This is where testers (or quality engineers) bring their game.
@James – this almost feels like the bulk of “uninteresting” tests have been automated and a perfect example of where exploratory testing could find “interesting” issues. But I’m not sure I have a good understanding how the testers fit in. I also feel that this is an example of where testing is getting “pushed” in general. I know at my company, the feeling is automation is the solution to testing. What I’m reading here reinforces that attitude.
Ken
@Joseph – I agree that CI is a useful and productive method for developing software. However, Timothy doesn’t refer to CI but to CD (Continuous Deployment). I read that as to mean continuous monitoring of the source repository, and upon change, compiling, building, testing and deploying the code into production. While that may sound cool, I think it’s a bit much and doesn’t give the testers an opportunity to discover defects before the code is deployed to production.
On the flip side, IMVU accidentaly managed to get free testing on their application plus testing advice/consultation. We’ll wait and see if they apply any fixes to the bugs you or Michael found!