Transformation Toolkit

Videos

Keep in mind that a requirement may have only partial coverage because not all possible test scenarios are covered by the test cases. For example, you may have identified 10 test scenarios, which includes the happy path, negative path, edge or corner cases, and multiple data combinations, but only create and execute test cases for 8 of them. Then the coverage for that requirement is only 80%. Expanding on the previous example, if in there are 100 requirements and 50 requirements have 80% coverage, 40 requirements have 90% coverage and 10 requirements have 0% coverage, then the total coverage would be, (50 x 80%) + (40 x 90%) + (10 x 0%) / 100 = (40 + 36) / 100 = 76%.

False positive rate is calculated by dividing the number of test cases that failed for reasons other than a defect by the total number of test cases executed. As it may be difficult to measure exact numbers, this metric can be estimated.

Efforts savings from Automation can be translated to costs savings by multiplying the time saved by the average cost of a manual testing resource. This amount of cost savings can be useful for promoting the benefits of automation and help drive conversations around resource and tool investments.

Imagine there are two requirements for a banking website. Requirement one states that users can transfer funds between their accounts, and Requirement two states that users can look up the closest bank branch or ATM.

Let’s first assign the requirements’ Damage and Frequency Class with a value of 1 to 5, one being low and 5 being high. For Requirement one, you may assign a value of 5 to the Damage Class, because the bank will face major fines if a money transfer fails and the user loses money. You may assign a value of 2 to the Frequency Class because the transfer functionality isn’t used that often. With these two numbers, we calculate the Requirement Weight for Requirement one by raising 2 to the exponential of the damage class and adding that with raising 2 to the exponential of the frequency class. In this case, it will be 25 + 22 = 36.

For Requirement two, you may assign a value of 1 to the Damage Class because failing to find a nearby branch or ATM isn’t that damaging to the business. You may assign a value of 3 for the Frequency Class since looking up a branch location is relatively used more often than making monetary transfers. Using these two numbers, the Requirement Weight for Requirement two would be 21 + 23 = 9.

Once all the Requirement Weights have been calculated, we can calculate the Requirement Risk. For Requirement one, the Requirement Risk is its Requirement Weight, 36, divided by the total Requirement Weight (36 + 9), which gives us 36 / (36 + 9) = 80%. Requirement Risk of Requirement 2 is 9 / (9 + 36) = 20%.

The Requirement Risk can also be seen as the business risk impact of the requirement. Note, that the sum of the Requirement Risk of all the requirements should always add up to 100%.

Next, is the Test Coverage calculation. If Requirement one has only 3 out of 5 test cases being tested than the test coverage would be 3 / 5, making the Test Coverage 60%. For Requirement two, if there are 2 out of 5 test cases being tested, then the Test Coverage would be 2 / 5 = 40%

Requirement Risk Coverage can now be calculated by multiplying each requirement’s Requirement Risk by the Test Coverage. For Requirement one it would be 80% x 60% = 48% and the Risk Coverage for Requirement two is 20% x 40% = 8%.

And finally, we sum up the Risk Coverage of all the requirements to get the Total Risk Coverage. In this example there are only 2, we have 48% + 8% = 56% total Risk Coverage for our banking website. That means that the current test suite is only testing 56% of the business risk of this application.

Automation Execution Rate is measured by dividing the number of automated test cases executed in a certain time period by the total number of automated test cases. For example, if your total automation suite is comprised of 1,000 test cases, and you execute 100 on a daily basis and the other 900 on a weekly basis, then your test execution rate is 10% per day (100 / 1000) and 100% per week (100 + 900 / 1000).

Get started with the Maturity Assessment