We all know the story: DevOps is quickly becoming the methodology of choice for software development teams (it hit 74% adoption in 2017) and it’s enabling those who adopt it to release new software faster and more often (McKinsey finds it can help teams reduce the time to move from code to production by as much as 74 days). And as CIOs face increased pressure to reduce time to market, the efficiency that DevOps brings becomes extremely valuable (IDC predicts that in 2018, CIOs will take agility to the next level using DevOps).
As often as the DevOps story gets told, the question of where QA activities fit in gets asked just as often. It makes sense — testing is typically viewed as a bottleneck, so how could the reverse also be possible? The fact that is still the mindset at many organizations presents a significant opportunity for those who are willing to see things differently.
Testing is all about assessing release readiness. If testing occurs early and often, this feedback is consistently available, meaning when issues are found, they can be resolved quickly. In this scenario, testing can be a significant boon to achieving the release frequency DevOps prescribes. In other words, prioritizing software quality can help organizations get better products into their customers’ hands faster. For many successful DevOps organizations, tapping into testing’s potential has offered an advantage that competitors have overlooked.
Five Major Challenges of Testing in DevOps
If it were easy to tap testing’s hidden potential and achieve DevOps transformation, wouldn’t everyone have done it? The truth is, there are several hurdles that teams must clear before they can operate in true DevOps fashion. Some of the most common challenges include:
- Lack of test automation coverage: Many testing teams are still primarily manual shops. One of the fastest ways to accelerate testing is to automate repetitive, manual testing tasks. But it’s not as simple as acquiring and mastering a new set of test automation tools. It’s critical to develop measurable goals and come up with a plan to transition existing manual tests to the new, automated paradigm. Making this shift correctly means manual testing must also pivot to focus on strategic, high-value areas. It’s as much a new mindset as it is a new toolset.
- Lack of visibility: As the level of test automation increases, testing teams often lose oversight and visibility because test management and reporting is isolated within each tool where it is performed. Without a holistic view of all test runs and results, it’s difficult to assess release readiness quickly, or ensure any issues found are prioritized appropriately. Establishing methods for documentation and reporting up front can help, but a tool that centralizes management, orchestration and reporting across open source test automation tools is even better.
- Limited traceability: Traceability can get lost in translation as teams transition to DevOps. Requirements evolve, code changes often and testing occurs frequently and often more disparately. One way to overcome this challenge is to version tests with code that’s being developed so that everyone can see how tests evolve alongside code changes. But the best way to ensure traceability in a DevOps environment is to enable collaboration, transparency and linkage across teams and tools. Testing must be evaluated alongside requirements, user stories and issues to adequately assess release readiness.
- Inability to maintain a single source of truth: When it comes to testing, no matter what methodology you follow, you need a single source of truth to house and share test cases, assess progress and eliminate data duplication. The pace, number of players and variety of tools involved in a successful DevOps release all complicate this imperative. Centralize testing up front to enforce standardization and enable teams to collaborate on requirements, evaluate defects and understand release readiness.
- Difficulty tying value to the pipeline: Finally, many teams face challenges tying business value to what’s in the pipeline. But being able to articulate value is important to getting approval, and when teams get this right, they typically move faster. Resolving this problem comes down to pairing what’s known as “agile upstream” activities, including Scrum, Kanban, daily standups and backlog planning, with “agile downstream” activities, including continuous integration, deployment and automated environment configurations. Without these two sides working in tandem, one group can outpace the other. Getting it right creates a seamless flow powered by a feedback loop that enables teams to move faster and demonstrate value more easily.
Maintaining a Culture of Quality
Once you overcome the five challenges listed above, you’ll be well on your way to a testing practice that is optimized for DevOps. But the work doesn’t stop there. Testing and technology leaders must continue to stress the importance of quality on a regular basis, across the organization. One of the best ways to do that is by communicating the team’s progress, sharing successes and becoming a resource to other teams across the organization. Building and maintaining a groundswell of support can help ensure continued investment in testing.
To effectively establish a culture that recognizes the importance of quality, leadership should regularly be championing testing’s work with both the business and product teams. If these stakeholders don’t understand the objective, their role in it and what it takes to get there, testers may end up with so much work that they can only focus on delivery, with little time left for larger conversations about quality. These teams need to understand that while devoting more time to quality might temporarily reduce time spent on product development, it will enhance the overall product and increase customer satisfaction. And that’s something everyone wants. Given these benefits, it’s not surprising that injecting testing earlier in the software development lifecycle is a priority for 30 percent of executives surveyed in the 2017-18 World Quality Report.
Getting developers on board is the next biggest challenge, and doing so requires focusing on their pain points and highlighting what’s in it for them. For developers, that usually has to do with fewer weekends and late nights. Start by measuring cycle time, which can help you measure the number of hand-offs and time spent on back-and-forth as defects are found. You can then share how much testers can reduce that time by finding defects up front.
How Two Companies Tapped Testing’s Hidden Potential
So how exactly does all of this look in action? Consider the case of Guardian Life Insurance, a 158-year-old U.S. insurance and wealth management provider that began making some big changes about five years ago to digitize its business. The business recognized testing as critical to the success of these efforts and invested in a modern test management platform so that testing could be integrated into agile processes and quality could improve. Following the implementation of agile test management across hundreds of projects, Guardian has significantly improved test coverage and cut test cycle times in half.
And Guardian isn’t alone. After modernizing testing and scaling test automation, EdgeVerve now tests 60% faster, on average, and has cut the rate of production defects in half.