Testers need to be thorough but risk being labeled as a bottleneck that delays release dates — particularly if testing uncovers bugs late in the development lifecycle. Testing bottlenecks represent one of the biggest risk factors in delivering high quality software on time. But understanding their causes and effects can help streamline QA processes and speed application delivery.
An assessment of where bottlenecks commonly occur can identify tools, processes or development area dependencies that keep the team from delivering against expectations. It can uncover faulty test suites that cause delays or turn out a higher number of defects than other parts of the workflow. Or it can highlight that testing has been left behind in an organization-wide effort to modernize software delivery.
Pinpointing testing roadblocks
Many businesses are transforming development and delivery practices to get software to market faster. But as DevOps.com points out, unless testing is modernized in tandem, it’s almost inevitable that testing will not be able to keep up with modern delivery cadences. The testing tools and processes architected for traditional months-long release schedules simply don’t fit modern delivery cadences, which require immediate quality feedback with each new build. If you have a slow testing process standing between highly-accelerated development and operations processes, there’s just no way that you can achieve the desired delivery speed.
Many companies have already recognized this — and they’ve catalyzed their digital transformation initiatives by modernizing their testing. By re-examining and re-inventing software testing across their organization, they went beyond removing the testing bottleneck to drive the positive user experiences that can now make or break a business.
Tricentis customer Merck sought out top development talent to advance its digital transformation initiatives. When some of the new developers started quitting, (now retired) Merck CIO Clark Golestani wanted to know why. He learned that the No. 1 reason was the company’s testing practices. Developers were frustrated because the company was still using traditional testing practices within Agile development processes — and developers believed it was undermining their hard work and dedication. (And you can imagine how frustrated testers must have felt.)
In response, Merck launched a full-scale testing transformation. Ultimately, the testing organization implemented agile testing practices and re-architected the toolset to scale test automation and enable rapid feedback. The result was dramatically reduced attrition and faster delivery with superior levels of quality.
“If it weren’t for the testing organization driving that change, they would have been an inhibitor,” Golestani said. “We would have never been able to release the types of products we do, and the technology would never work the way that it needs to. It’s absolutely essential.”
Metrics that can help you pinpoint testing bottlenecks
The earlier you start to measure data that tells a story about your process and performance, the better chance you have of pinpointing the cause of testing bottlenecks and transforming tools and processes to eliminate them. Whether you are on a weekly or monthly sprint release, the following standard metrics apply.
- Defect aging:
Typically assigned upon discovery, defects should appear on a list for each sprint that summarizes the issue, notes the dates of detection and reports the status. Resolved issues move to a defect archive at the end of the sprint, but glitches or missed requirements that need attention remain on the list. For those familiar with Agile methodology, defect aging keeps teams accountable and encourages open communication between teams that may need to collaborate to solve problems. Sprints remind all stakeholders of open tickets and accelerate movement to resolution, back to requirements gathering, to a future sprint or to the backlog.
- Defect open/close trends:
A cumulative view of defects opened vs. closed over time supports continuous process and quality improvement with trend data for each release. Timing parameters can highlight the rate of defect activation or termination for specific sprints.
- Open defects by category:
Open defects by category provides additional understanding of the pain points in your software development process and directs managers to focus attention on these ruts in the road. Useful categories include severity, priority to the business and defects by module. New approaches to design, code fixes or other improvements can move software through planned testing to an on-time delivery.
- Cost per defect:
The impact of defects and the timing of their discovery both influence what it costs to fix them. You can resolve defects found in requirement specifications, for instance, before architecture planning, well in advance development. Similarly, you can replace errors in design with re-issued documentation. But one defect in the requirements may be replicated in several places in the design and code. For that reason, defects uncovered in construction or in user acceptance consume more hours to remove and require repeat testing.
Velocity measures the actual effort your team expended, measured in time spent on the project, vs. how long you estimated it would take. Velocity has an obvious impact on meeting release timelines. Paired with function-specific metrics, velocity can improve with additional tools, consulting or reinvented workflows.
- Effort variation:
Not unlike velocity, effort variation helps quantify actual effort vs. planned effort. The metric, observed sprint to sprint, can help keep current projects on budget and on time. It also can inform planning of subsequent software development.
Learn from past experiences, improve future plans
Intelligent, relevant testing metrics drive learning about your software development lifecycle and help you control outcomes. With action-oriented data, you can stop reacting to issues and start preventing them and the bottlenecks they introduce.