Everyone exposes themselves to risk. Risk is an omnipresent part of every human endeavor, regardless of whether you are speeding on the highway or just taking a breath of fresh air. Testing is no exception.

Any software tester can tell you that risk and testing are related, but there is still very little clarity as to what risk-based testing actually is. Testing literature is replete with false statements, incomplete, and ambiguous explanations on the topic. In my experience, “risk-based testing” is often just a buzz-phrase with little substance behind it.

When I ask the question “what is risk-based testing”, I often receive the unsatisfactory response that risk-based testing is “testing based on risk”. While that answer isn’t wrong, it tells me that either “risk” or “testing” is not well understood in the testing community. Since the latter option seems highly improbable, our goal is to understand what “risk”, or more specifically business risk, means in the context of testing.

Risk Definition

Let’s start by looking at business risk in order to understand what it causes.

Given the ubiquity of the general term, it is surprising how little consensus there is about its proper definition. Risk is incorporated into so many distinct disciplines, from insurance to engineering, that it is often defined in totally different ways by each one.

This brings us to an important conclusion: the meaning of risk varies widely between distinct professions. In other words, in each of these professions, the risk tells a different story. If we want to work out a universal definition to the term, we need to take a step back.

Download Whitepaper

In its most fundamental form, the term risk quantifies the potential of losing something of value.

Let’s take this studiously vague definition as a starting point. We can derive a simplified meaning of “business risk” from this rough definition by stating that business risk quantifies the potential of losing something of value from a business perspective.

This statement has a clear negative undertone. The meaning of “business risk” may be rewritten to say: something that quantifies the potential of causing a negative impact on a business. Any kind of negative impact causes harm by default, and as a consequence, immediately leads to loss of something desirable (also known as damage).

With that, we can take our definition one step further: business risk quantifies the potential of causing some yet unknown damage to a business.

Now let’s get really nerdy and work out what the term potential means. “Potential” defines something possible but not yet actual. With respect to the business risk definition derived so far, this term loosely defines an ability to cause some damage on the business. In terms of software testing, it is obvious that something possible, but not yet actual, that has the ability to cause some damage on the business, stems from a software error.

Error!? Again, such an ambiguous term. Words such as bug, defect, fault, failure, anomaly, error and mistake, to name a few, usually cause more confusion than clarity in the testing community. Have you ever counted the number of discussions and debates on the internet about the distinctions between these terms? Infinite! The principal reason is that a common objective definition of each of them doesn’t exist. Knowing that this may not be the universal industry standard

A software error is a programmatic manifestation of a mistake made by a programmer, causing discrepancy between the actual and the intended behavior of the software.

A software error therefore naturally causes damage to a business, since the software product containing the errors doesn’t behave as intended by itsstakeholders. In classical product development, a stakeholder is anyone who could be materially affected by the implementation of a new product or by a change made to an already existing product. (For the sake of this argument, stakeholder will also include customers and users.)

This implies that “business” must refer to anything that could be materially affected by that error. From this we can safely conclude that business risk quantifies the overall damage to a business arising from some yet unknown number of possible distinct errors.

Ok, let’s regroup. What are we talking about again? We are trying to define the business risk associated with a particular requirement. According to (1) a requirement, in its most general form, is defined as follows.

A requirement is a condition or capability that must be met or possessed by a system or system component to satisfy a contract, standard, specification, or any other formally imposed documentation. Hence, a requirement represents a condition or capability required by a stakeholder to solve a problem or to achieve an objective.

That is a lot to process. Using that definition, a software error causes a condition or capability required by a stakeholder to not be met or possessed. The business risk associated with an individual requirement quantifies the overall damage on the business, arising from distinct errors that may live inside the requirement’s functional and non-functional spectrum. Distinct, or independent, errors are usually associated with different severities. Each individual error contributes its individual part to the overall damage, meaning that each error has a different negative impact on the business.

Now we have a problem. The total number of possible distinct errors, each with a different impact on the business, hidden in a particular requirement, is virtually impossible to assess. How can you approximate an estimate with any accuracy? The only way out of this dilemma is to assignprobability to these errors, and then to replace the term “damage” with “potential damage”.

Your potential damage can be found by using the law of large numbers. This mathematical theorem, allows us to approximate the total number of detections of one error, by its corresponding error probability. In this context, error probability defines the probability of occurrence of a particular error within each call of a particular requirement. This is why probability is fundamentally intertwined with the definition of risk. It’s simply due to a mathematical theorem.

At a glance, it may seem strange to think about a call of a requirement. What does that even mean? To make sense of it you need to think about the requirement’s underlying functional and/or non-functional spectrum (i.e. its functions and/or features) that characterizes a specific requirement. Simply stated, a requirement’s call is just another word for the execution of the entire spectrum that underlies an individual requirement. The number of calls of a particular requirement within a certain timeframe is the requirement’s frequency of usage.

Now take a deep breath and hang in there. We are done here in a moment, since all of these thoughts can be elegantly summarized by the following single sentence:

The business risk associated with an individual requirement quantifies the overall potential damage on the business that is caused by multiple distinct errors in each execution of the requirement’s (non)-functional spectrum.

From here, we conclude that business risk associated with an individual requirement has to be a result of the potential damage and the frequencyof usage of the requirement itself.

Congratulations, you made it through.

Almost!

We have just explored what the term business risk means, but not how to measure it. Peter Drucker, a prolific management writer, is often erroneously cited as the creator of the phrase “If you can’t measure it, you can’t improve it”. Exactly where the phrase came from is unknown, but the idea remains valid. We need to be able to measure that business risk, to improve our testing. We can’t be satisfied with the fact that business risk is just a function of the potential damage and the frequency of usage (referred to from now on as business risk components). We need to figure out what the business risk function looks like.

Continue Reading the Whitepaper

Find out about:

Risk Perception

Risk Derivation

Risky Pitfalls

Conclusion