My son is ready to send the manuscript of his novel to publishers. It’s time to see what the interest is. In other words, we are going to beta on it. He made this decision tonight.
What is the quality level of his manuscript? There is no objective measure for that. Even if we might imagine “requirements” we could not say for sure if they are met. I can tell you that the novel is about 800 pages long, representing well more than 1,200 hours of his work alone. I have worked a lot on editing and review. The first half has been rewritten many times– maybe 20 or 30. It’s a mature draft.
The first third is good, in my opinion. I’m biased. I’ve read the parts I’ve read many many times. But it seems good to me. I cannot yet speak about the latter 2/3 because I haven’t gotten there yet. I know it will be good by the time we’ve completed the editing, because he’s using a methodical, competent editing process.
Here’s my point. My son, who relies on me to test his novel, has not asked me to quantify my process nor my results. I have not been asked for a KPI. He cares deeply about the quality of his work, but he doesn’t think that can be reduced to numbers. I think this is partly because my son is no longer a child. He doesn’t need me or anyone else to make complicated life simple for him.
How do you measure quality?
Gather relevant evidence through testing and other means. Then discuss that evidence.
That’s how it works for us. That’s how it works for publishers. That’s how it works for almost everything.
Who can’t accept this?
Children and liars.
But my company demands that I report quality in the form of an objective metric!
I’m sorry that you work for children and/or liars. You must feel awful.
I have often said something like “We found a hundred bugs!” Lots of people have heard me say it. Statements like that are very valuable to me. But we should ask some vital questions about them.
Consider Raisin Bran cereal. If you lived in America and weren’t in solitary confinement during the 80’s an 90’s you would have seen this commercial for Raisin Bran at some point (or one like it):
Two scoops of raisins!
Huh? Two scoops of raisins? What does that mean?
Perhaps the conversation went like this:
“I want my Bran to have MANY raisins!” barked Boss Kellogg.
“But, Mr. Kellogg, we already include nearly one full standard scoop.” replied the Chief Cereal Mixer. “No one has more raisins than we do.”
“Increase to maximum scoop!”
“But sir! that would violate every–”
“TWO SCOOPS! And damn the consequences.”
“The skies will be black with raisins!”
“Then we shall eat in the shade.”
I doubt anything like that happened, though. I suspect what happened is that somebody mixed some raisins with some bran flakes until it tasted pretty good. Maybe he adjusted it a little to optimize cost of goods (and perhaps they adjust the bran/raisin ratio as cost of goods change). Later, I bet, and completely unrelated to the engineering and manufacturing process, Kellogg’s advertising agency decided to create the impression that customers are getting a lot of value for the money, so they invented a distinguishing characteristic that actually makes no sense at all: an absolute measurement called a “scoop”. And began to speak of it AS IF it were meaningful.
The reason the measurement makes no sense at all is that the “Two Scoops” slogan was pasted onto boxes of substantially different sizes. But even if the measurement makes no sense, the pretentious claim makes a lot of sense, because we humans don’t think through the rational basis of measurements like this unless we are A) rather well trained, and more importantly B) highly motivated. So our unconscious lizard brain says to itself “two means yummy. two means yummy. means two yummy. yummy two…”
At some point, someone (an intern, perhaps) may have asked “But are there actually two scoops of raisins in those boxes?” and the answer was much laughing. Because it could be argued that if there are at least two raisins in the box, then there are two scoops of raisins in the box. It could be argued that if there is one raisin in the box and you used two scoops to measure it (“measure twice and cut once”) then there are two scoops of raisins in the box. If you make up your own measuring unit, such as, say, “scoop”, you can go on to make any other claim you want. This is exactly the point of Jerry Weinberg’s famous dictum “If quality doesn’t matter, you can achieve any other goal you want.”
We have a real problem in testing, and no good solution for it. We are supposed to report the ground truth. Concrete reality. But this turns out to be a very difficult matter. Apart from all the problems of observation and interpretation, we have to summarize our findings. When we do that we are tempted to use scientific tropes (such as nonsensical measurements) to bolster our reports, even when they are poorly founded. We are often encouraged to do this by managers who were raised on Kelloggs commercials and therefore confuse numbers with food.
Let’s look once again at the Raisin Bran situation and consider what might be the reasonable communication hidden there:
Maybe “two scoops” is intended to mean “ample” or “amply supplied with raisins.” In other words they are saying “You won’t regret buying our Raisin Bran, which always has enough raisins for you. While you’re eating it, we predict you will hum the ‘two scoops of raisins!’ song instead of calling a lawyer or becoming a cereal killer.”
I think there’s a scale built into all of us. It’s a comparative scale. It goes like this:
“OMG! That must be a record!”
This scale is a bit of a mess. The italicized values move around (e.g. maximum possible may be not enough in some situation). The others although fixed relative to each other, aren’t fixed in any way more definite than their ordering. The scale is highly situational. It’s relative to our understanding of the situation. For instance you might be impressed to learn that the Colonia cable ship, which was the largest cable ship in the world in 1925, could carry 300 miles of cable in her hold. If so you would be very easily impressed, because I just lied to you… According to that article it actually could hold 3,000 miles of cable. (However, bonus points if you were thinking “what KIND of cable?”)
What I do with bug numbers, etc.
I want you to notice my first paragraph in this post. Notice that every sentence in that paragraph invokes an unspecified quantity.
“I have often…” Often compared to what?
“Lots of…” Lots compared to what?
“Very…” Very compared to what?
“Vital…” Vital compared to what?
You could say “He’s not saying anything definite in those sentences.” I agree, I’m not. I’m just giving an impression. My point is this: an impression is a start. An impression might be reasonable. An impression may make conversation possible. An impression may make conversation successful.
Most engineering statements like this don’t stand alone. Like flower buds, they blossom under the sunlight of questioning. And that’s why I can’t take any engineer seriously who gets offended when his facts are questioned. They cry: “Don’t you believe me?” I answer: “I don’t know what you mean, so belief has no meaning, yet.”
So, as a professional tester who prides himself on self-examination, I am ready for the probing perspective question that might follow my attempt to send an impression: “compared to what?” I am ready for the data question, too: “what did you see or hear that leads you to say this?”
I strive (meaning I consciously and consistently work on this) to be reasonable and careful in my use of qualifiers, quantifiers, quantities, and intensifiers. For instance, you will notice that I just used the word “reasonable”, by which I intend to invoke images of normal professional practice in your mind (A LOT like invoking the image of two healthy reasonable scoops of delicious raisins).
One important and definite thing that is accomplished by this relatively loose use of language is that it allows us to talk to each other without bogging down the conversation with ALL the specifics RIGHT NOW.
Kelloggs used the method mostly to trick you into buying their bran smothered raisin products. They didn’t have any reasoning behind “two scoops.” But we can use the same technique wisely and ethically, if we choose. We can be ready to back up our claims.
For Bugs: If I tell you I “found X bugs!!” in your product, the number of exclamation points indicates the true message. An exclamation point means “remarkable” or “lots.” If I tell you I found a lot of bugs in your product, I mean I found substantially more than I expected to find in the product, and more than a reasonable and knowledgeable person in this situation would consider acceptable. And by “more” I don’t mean quantity of bug reports, I mean the totality of diversity of problems, impact of problems, and frequency of occurrence of problems. The headline for that is “lots of bugs” or maybe I should say “two scoops of bugs!”