Responsible Quality Engineering

Let’s talk about responsible quality engineering (RQE).

Quality engineering is not testing, although it may require testing. Also, most “quality engineers” don’t do quality engineering. The people who do quality engineering are developers. Testers support that process by analyzing the quality of the product and providing feedback to developers as part of the quality engineering process. I want to introduce you to a way of thinking about quality engineering that I call the Quality Engineering Tetrad.

Let’s start with first principles.

• Quality is value to some person (who matters). Quality is always a judgment made in context– an assessment, rather than a simple measurement. Quality is a relationship between a person and a product. In a world without people there would also be no such thing as quality.
• Quality engineering is the systematic process of “creating quality.” To create quality is to create a product that is judged as valuable by some person who matters. You do this by engineering. But “quality engineering” is not the same as engineering. When we speak of quality engineering, as such, we are referring to two big things: systematically satisfying someone who matters (other than yourself), and engineering a reliable and economical process by which the product comes into being. Quality engineering matters because when it comes to complex systems, quality doesn’t happen automatically.
• A responsible process is a process for which some competent human is accountable. By “competent human” I mean a natural person with the capacity, commitment, and skills to have a reasonable chance of assuring good work and is reasonably prepared to explain, defend, and otherwise answer for the quality of that process. Since an AI tool cannot be accountable for anything, it cannot enact or embody a responsible process.
• In responsible quality engineering, there is at least one provider, a person who creates the quality product, and at least one receiver, a person who experiences and enjoys that product. Note that if the provider and receiver are the same person, you don’t need the discipline of quality engineering. Just do what you want until you feel successful.

Quality itself doesn’t necessarily require a human provider. You can look out upon a beautiful sunset and say “that sunset has quality!” It’s a high quality vision to behold. In that case, there is no provider, only the receiver, who is you. Quality exists here because you have judged it into existence. The sunset is consistent with what makes you comfortable and happy, compared to the searing sunburn you might receive at midday. But this post is not merely about quality, but rather quality engineering. Moreover, it’s about responsible quality engineering. RQE spans the whole process from provider to receiver.

To engineer quality, the provider must conceive what is good (imagination), communicate what they will build (specification), make it real (delivery), and then it must be received by a receiving person who feels its effects (experience). The receiving person also must have an understanding of the product (imagination) and receives some explanation and description of it (specification).

These four aspects of the product must be intentionally aligned, in the judgment of people who matter, for us to say that the quality of that product has been “engineered.” See the diagram below:

Notice that the four aspects of the product are presented as a Venn diagram. Where all four overlap, we have engineered quality (as opposed to accidental quality). That’s the green section. We want to maximize that green part. Testing is the process of assessing that alignment. In any case where all four don’t overlap, we have some sort of breakdown or gap in quality engineering (although not necessarily a breakdown in quality itself, since you can have accidental quality).

Notice that all the subsets can overlap without necessarily meaning that the quality is good. The overlapping sets just mean that the receiver is experiencing what the provider meant to deliver. There is still the issue of whether that is what the receiver wants! That’s where the text in the corners comes in. While striving to deliver purposefully, we also have to validate that our idea of good matches that of our clients.

And we must do all that reliably and at a reasonable cost.

Providers vs. Receivers

In the commercial world, you cannot disconnect quality from responsibility. This is because our purpose in developing products is to fulfill the needs of the business, which include serving and satisfying some customer within the bounds of some contract or business relationship. There are well understood concepts in law and ethics which help us think about responsibility. Among them are:

Duty of care: “the legal responsibility of a person or organization to avoid any behaviors or omissions that could reasonably be foreseen to cause harm to others.” (from legaldictionary.net)

Reasonable: “just, rational, appropriate, ordinary, or usual under the circumstances. In law, it is a flexible standard used across many contexts, including reasonable care, cause, accommodations, and doubt in criminal trials.” (from www.law.cornell.edu)

Responsible: “A person or body who can be held accountable for carrying out a duty is considered responsible.” (from www.law.cornell.edu)

Contract: “An agreement between parties, creating mutual obligations that are enforceable by law. The basic elements required for the agreement to be a legally enforceable contract are:

• Mutual assent (offer and acceptance)
• Consideration (something of value is exchanged)
• Capacity (e.g., minimum age, sound mind)
• Legality (lawful purpose)”

(from www.law.cornell.edu)

Representation: “A statement of fact made with the purpose of getting someone to become party to a transaction or contract. The presentation of fact may be by words or by conduct and it should induce a person to act.” (from definitions.uslegal.com)

These concepts are important, because quality engineering is the opposite of mere trust. If you trust, you don’t need to engineer quality. You need it when you want to compel quality into existence, and you want to know it is there. This matters in a world where people use AI tools to get work done. We tell AI to build software products for us. It’s only natural to want to tell the AI to assure quality in those products. But AI cannot do that, because that would require AI to take responsibility. Unlike a person or a company, a tool cannot be held accountable for its behavior. It has no capacity to enter into a contract. If a tool misbehaves, the fault lies either with the operator or the provider of the tool.

When Imagination Doesn’t Match Specification

This means we aren’t saying what we mean, or at least we aren’t saying everything we mean. It also includes not saying what we want. This is not necessarily a problem. Plenty of things go without saying. Still, quality engineering is predicated partly on our ability to know our minds and express ourselves, and partly on our ability to develop better ideas and specs over time.

When Imagination Doesn’t Match Delivery or Experience

Even if fully and properly specified, and even if the delivered product matches that specification, it still may not fulfill the desires or needs that motivated the development of that product. This can happen because needs and desires do not remain fixed over time. We learn, we grow, we get new ideas. This can also happen because the world changes around us, or because there were multiple people involved and they were forced to make compromises in the specification, so that no one is getting everything they wanted.

This is especially a challenge when we are put under pressure to specify a product before we’ve done enough research to develop those ideas. In Agile projects, we consider it normal for needs and wants to evolve. That’s why we iterate. If a product matches its spec but not what we want, this normally results in an enhancement request.

When Specification Doesn’t Match Delivery or Experience

This is what we call an ordinary bug. In fact, when the spec doesn’t match the delivered product, it is not just a bug; it’s a defect. (In RST, a bug is anything about the product that threatens its value. We call things bugs when we suspect that value is threatened. But when something is definitely wrong with a product, based on a direct contradiction of claims we are making about it, then it is fair to say there is a defect.)

When Delivery Doesn’t Match Experience

Even if a correct product is delivered, it is possible that the user does not receive the experience that the developers intended. This can happen because of problems with hardware or software platforms that are outside the control of the developer, or because of constraints in the user’s environment or attributes of the user themselves that were not anticipated by the developer. It may also happen because of foreseeable misuse.

Iterative and incremental delivery, coupled with a strong feedback loop, helps a lot with this problem. Also, making a product configurable allows different users to tune it for their needs.