Speed is the new currency
It’s no secret that the past 20 years have seen a seismic shift in how software is being developed – and the pace of change has not let up. There is a constant influx of new tools and technologies, and customer expectations swiftly catch on, with the default level of expected software quality creeping up every moment. Pair that with increasingly complex architectures, requiring testing across multiple applications, layers and teams, and it becomes clear why testing often remains a bottleneck.
The market has proved that it favors organizations who can release new functionality quickly without compromising on quality, and business strategies have adapted accordingly. Organizations fully embracing DevOps and agile practices are seeing a 60 percent higher rate of revenue and profit growth, and are 2.4 times more likely than their mainstream counterparts to be growing their businesses at a rate of over 20 percent, according to a recent Freeform Dynamics survey.
But many businesses are quick to prioritize delivery speed, with the quality side of the equation getting pushed down the road. Oftentimes, testing becomes the last piece of the puzzle, even as other parts of the software delivery pipeline are modernized. The 2020 Gitlab Global Developer Survey found that despite the increased delivery speed that DevOps delivers, most DevOps teams encounter the majority of delays during the testing stage.
According to the 2019 Accelerate State of DevOps report by DORA, the number of organizations that are characterized as ‘elite’ nearly tripled between 2018 and 2019. These elite teams ‘automate everything’ by committing to continuous integration and delivery practices, the success of which hinges on effective test automation. According to the report, “smart investments in building up automated test suites will help make CI better.”
But as long as testing lags behind, businesses leave themselves open to risks they can’t quantify. They also negate any potential benefits of rapid innovation if they are unable to ensure the quality of new products at an accelerated pace. And there is mounting evidence that modernizing testing is critical to successfully accelerate software delivery. It’s time for testing to start engaging with the idea of transformation, and for quality leaders to step into the role of an agent of change.
The following guide shares our advice for spearheading testing transformation at your organization – from developing and sharing your vision for a modernized testing practice, to gaining support and driving widespread adoption across the organization.
Spearheading testing transformation: A three-step approach
Whether you are supporting an organization-wide transformation or looking to modernize one key aspect of the testing practice, your transformation effort could involve all or any of the following elements:
- Transitioning to Agile testing
- Scaling test automation
- Integrating with DevOps pipelines
- Improving enterprise app quality
- Supporting cloud migration
- Improving data integrity
- Any other aspect of software quality, whether it’s customer facing, internal or both
When most organizations begin to tune up their testing, they are trying to approach the optimal level of testing that offers an acceptable degree of risk coverage, without delaying delivery.
And while new toolsets and processes can help testing teams move in the right direction, when we’re talking about rolling out changes necessary for a testing transformation, new tools alone aren’t going to be sufficient. You’ll need a whole new approach, and a sustained change management campaign.
According to Dawn Hall, Test Manager at the International Air Transport Association (IATA), it all starts with strategy. “If your test strategy is wrong, then the tool can’t fix that,” she says.
Learn how Dawn Hall quickly scaled test automation in the following video.
In January 2018, she kicked off a significant testing overhaul, transforming primarily manual process into a lean testing approach to better align with the organization’s evolving business model. Her goals included reducing the number of test phases, reducing production incidents and reducing business effort during testing – all of which contributed to the rollout of DevOps pipelines.
If you’re reading this, you’re probably already working on one of your organization’s most innovative teams. And you can probably envision the profound impact a modern approach would have if it were extended across the organization. But how can you roll out the changes from the first team out to 10, 20, or 100 more? You need a scalable approach.
1. Creating your vision
Align with the business
Successfully modernizing testing requires alignment with broader business strategies, so the first sanity check you should do is to ask yourself whether that’s the case, and how to make that connection clear. You will be smoothing out the road ahead for your proposed testing transformation if its contribution to a broader organizational initiative is undeniable – whether that’s scaling DevOps, supporting cloud migration, or accelerating testing to support new products, business channels or revenue streams.
The journey can’t be clear if your vision isn’t clear
A strong vision will help you build a clear plan with blocks of work that can be broken down into discrete tasks. You can then work back from the tasks and see how each step will contribute to the end goal. And while it’s often the case, testing transformation doesn’t have to follow the adoption of agile or DevOps elsewhere in the business.
At a large financial technology company, it was the Test Engineering team which led the DevOps initiative and sparked an enterprise-wide transformation. Inspired by a talk on DevOps at an industry event, the VP of Test Engineering and Delivery Acceleration began seeking advice from industry leaders and peers to understand DevOps release pipelines, and the test strategies needed to support them.
As the development organization progressed with the agile transformation, Jones and his team created a cross-functional QA center of excellence and built up to delivering software to their customers via 15+ operational DevOps pipelines, reducing testing timelines, production defects, and deployment time from 14 hours to 4 minutes for certain applications.
What does a clear vision look like?
Your vision could be for a complete overhaul across testing functions, or it could be pointed towards one specific aspect of the testing practice. Take Toyota, for example, whose modernization of load testing has had a significant impact. They set up a Testing Center of Excellence (TCoE) whose vision was to: “Provide, innovate, and collaborate solutions to drive built-in quality.”
Toyota has a long-standing commitment to quality, and when their TCoE began evaluating if and how to replace its legacy load testing tools, they boiled down the main drivers to move ahead:
- Testing needed to shift left to meet the demand that load testing be done earlier in a development cycle, when any issues with performance are cheaper and easier to fix
- On-premise resources were costly to maintain and limited the amount of parallel load testing which could be carried out
- Previous tool could not meet the needs of agile and DevOps and therefore had low adoption rates among developers
- Testing was chronically late in the development cycle, creating stressful timelines as issues had little time to get fixed before code needed to be pushed into production
In a presentation at Tricentis Accelerate San Francisco, Toyota shared how they rolled out the new, cloud-based load testing solution to 40 separate teams.
When IHG decided to federate testers from their TCoE back into developer teams, the created a “Global Quality Engineering Ambition Statement” to drive home the intent and goal behind this potentially disruptive change. The statement established testers’ commitment to solving the team’s needs, disseminate industry best practices, optimizing test practices, and implement continuous testing to ultimately improve product quality:
“Continuous Quality at Speed.— IHG, Global Quality Engineering Ambition Statement
[Our mission is to] partner with, and create sustainable value for, Commercial & Technology teams by challenging the status quo and brining innovative ideas and technologies to address their Testing and Quality Engineering needs.”
Workshopping the idea
As you’re developing your vision, validate and workshop your idea with a wide range of people: Is it good? Will it hold up? Is this change actually necessary? Is it the type of change that we might need? As more people provide feedback, various paths will start to appear. Use these differing voices to figure out the best way your idea will take root.
Although you will likely begin with a small control group and will be planning small scale initiatives for a while, you don’t want your idea to peter out early, so don’t be afraid to have a big vision from the start.
Practice pitching your idea to different stakeholders
Prepare answers for the one question you will hear from every stakeholder you need on your side. They may not use these exact words or be as direct, but know what you’re going to say when you hear: “What’s in it for me?” This is a good exercise and helps you consider the impact of your transformation in concrete terms. Why is this change valuable for those affected? How is it going to affect the company? You might be crystal clear on the positive benefits, but have you considered potential pitfalls or challenges other teams might face?
For example, when speaking to developers, you could explain that a culture of quality means fewer late nights, working weekends or rushed deadlines to rewrite code where a bug was found late in the sprint. Paint a picture for them – you could calculate and show them the time they could have back if an issue was detected earlier in the development cycle.
Be mindful when first presenting your idea, to strike a balance: How can you communicate the essence of the idea without oversimplifying the setbacks or hurdles that may stand in your way? You will be eager to gain buy-in from those you speak to, but it will backfire if they later feel that you underplayed the growing pains.
Conversely, expect to receive some pushback, or hear about potential setbacks you may not have considered. Dig into why someone would resist or reject the proposal, and plan buffer time to account for hard or lengthy discussions. Major change always comes with some degree of growing pain, so you don’t want to shy away from this. Be open to the possibility of needing to reset and pick up your plan from the beginning, or an earlier stage. All of this will build resilience into your proposal.
2. Gaining first supporters
You may gain your first supporters as you practice and refine your pitch, giving you a base of supporters (or at least colleagues who are not hearing the idea for the very first time) as you try to gain enough momentum for your idea. As your idea hopefully begins to gain ground, keep the following points in mind.
If you are working in a highly regulated industry, the transformation may move slower – be patient when planning and garnering support. But the fact that the consequences of a defect in such a heavily regulated environment can be significant may strengthen your case. This article shares how adopting test-first methodologies can benefit teams that are working in regulated environments.
Focus on getting your first and second win. Then the word of mouth will spread.— Director of Testing at a Global Manufacturer
Learn how to effortlessly establish the value of testing
Be careful not to fall into the common trap of framing testing as ”finding bugs” or ”breaking the product.” Testers provide information and help to uncover problems that threaten the value or timeline of a product. Good testers can do this early in the development process. Testers don’t directly prevent defects, but their work can prevent uninformed and underinformed decisions.
Find a quick project and a get proof of concept under your belt
By collecting real data within your company, you will lend credibility to your idea, build transparency, and demonstrate what taking broader action might look like. This is especially effective if you are even somewhat able to address a pain point. You’ll need to line up a small team who can commit to trying out a small but tangible project for a sprint or two.
At Worldpay, QA Leader Sandra Baker understood that there would be different mindsets across the organization’s distributed agile teams. To facilitate the rollout of a new automation strategy, she arranged a roadshow to visit key teams in person to discuss their unique challenges, and demonstrate her team’s early successes. Baker shares the details of her approach in the following video:
Think about finding an ally – for example, a product manager who can help make the idea more concrete and present the idea in a language that resonates with how the leadership team thinks. One way you can get started is to consider what testing jargon you can make more accessible.
According to Robina Laughlin, who spearheaded agile testing transformation at Guardian, active and frequent communication with the broader organization was key to her success. “We spent a good 3-4 months talking about why we were doing this, what it meant, and the opportunity it would present for the team,” said Laughlin, who is Assistant VP of IT Quality Management at Guardian.
“When we brought a new change into workflows, we planned ahead to find good cutoff points in a spring/arc in our more active projects, so everyone knew when the shift would be made,” she explained.
Executives are people too
Similar to finding an ally, finding an executive sponsor is also a very powerful way to shore up support in those early stages of a project. Being able to answer: “What’s in it for me?” and “Why is this best for the business?” is crucial here.
According to Alex Kyriazis, Senior Manager in Testing at ANZ, executives will want to understand the overall benefit first, such as whether you’ve mitigated risks or provided enough coverage to deploy code with confidence. You need to explain your metrics in terms of the value it provides to them.
“Say you tell a C-level executive that you have 100,000 test cases which you automate every night, that you are now delivering software quicker, or that you’ve saved them a million dollars,” Kyriazis says. But for all the executive knows, “you may have cut a million dollars in the wrong places or managed to deliver defects into production faster. They will want to know: ‘So what?’”
Executives will often speak in terms of dollars and effort expended so if you can, find the hard numbers to support your idea – like cost incurred if all testing remains manual, team productivity metrics, or potential time savings across teams. Communicate the benefit of the change, and be upfront and realistic about the expected cost. With executive support, you may even be able to get fast-tracked through some common roadblocks, such as new training programs, or facilitating collaborative groups within your organization.
Once you have the main proposal outlined, contextualize your change in terms of customer satisfaction (cost, speed of updates, effectiveness), and tie it back to the vision. The clarity of that vision will help you here as individual tactics will almost certainly evolve as your journey unfolds, and having a clear vision will serve as a compass during the setbacks and revisions that happen over the course of such projects.
Communicate actions and results early
Give colleagues notice and try to minimize surprise – your small, early changes will probably mean changes for them too. On the plus side, early adopters tend to have plenty of opinions, so it’s easy to solicit feedback. You can also stress that projects are easier to adjust early on, so concerns are especially gratefully received at this stage.
According to Barbara Vlahos, an IT Manager in Global Quality Engineering at IHG, “There is no such thing as too much communication. Overcommunicate, it never hurts.” Vlahos, who spearheaded the initiative to adopt and scale agile, says she relies on multiple channels to communicate the change and its impact. “We now have town halls, newsletters, printed handouts, a user guide, an FAQ, and we make sure we invite everyone to training.”
Early results, even if they’re not outright successes, should not be hidden away either. Use early data points to address and incorporate learnings, or to share successes to reassure any stakeholders who may have had misgivings.
3. Building momentum and adoption
The team that first comes on board to drive the first changes will likely be led by someone who’s passionate about the topic, with handpicked enthusiasts making up the team. Unfortunately, the second, fifth, or tenth team that needs to pick up the change is unlikely to match that level of commitment, and you need to think about how to scale without stalling.
Have strategies ready in cases of lost momentum
One of the quickest ways to halt a promising change in its tracks is to get disheartened at the first failure. This may be a resistant department, inconclusive proof of concept, or lack of management support. Know that it is part of the process to stop and reassess, chart alternate routes, or simply go back a few steps, and it doesn’t necessarily mean that your idea is not worth pursuing.
Create a central place for members of the team to document lessons learned. As with the vision-building phase, asking for feedback from people in different roles, with different goals will create a more well-rounded reference when trying to understand why something was unsuccessful.
During IHG’s initiative to modernize test management for agile, they were prepared to ‘respond quickly and effectively’ with every new team which was brought on board. Before training was rolled out to each team, a pre-meeting was organized to learn their unique workflows, and to ensure they would not feel like a change was being forced onto them.
When they encountered an area of the business with a complicated use case, they arranged a new round of demos with Tricentis, where these teams could provide feedback and submit feature requests, involving them in the journey. By being transparent and addressing areas which were not working or meeting expectations, they continued to build on their transformation instead of having it grind to a standstill.
The metrics that will be most compelling will differ for every function or level of seniority you are talking to. Developers may want to know: How long does it take to move an item from one point to another inside of a sprint? How long does the release and the build take? How long does the testing take? While VPs of Engineering will be interested in risk coverage per release and improved efficiency for getting to market.
There may also be a difference in the metrics that will help those who are on the front lines of the transformation efforts, and those who only need to know how it’s going once a month. Be sure to consider metrics that will keep team members on the ground motivated, as well as the slower-moving ones which will capture the overall success of the transformation.
Think about telling a testing story – quickly set up the vision, share the status of the project/product, what the current vulnerabilities are, and which potential failures will matter most to the clients. Like many of the other aspects discussed in this paper, metrics are another aspect of the project you may need to revise along the way.
Adoption by building community. How can you make this change stick?
As your change enters a stable phase, think about building a community of practice which goes outside of testing and also invites business users, product managers, developers and so on. Create spaces for easy knowledge sharing among this group, whether it’s an active Slack channel, lunchtime sessions to share updates or the latest learning, or regular conference calls.
Nurture a small group of champions throughout the organization (often the early adopters) to continue monitoring uptake and spreading your cause. They could be involved in the community of practice above, lend a hand in keeping training documentation updated, or be an extra vector by which updates and ongoing improvements can be communicated.
Let your wins be the leadership team’s wins. This is particularly effective if you were able to gain an executive sponsor. Acknowledge the opportunity and support that was created because of leadership support. In organizational cultures where quality is a shared responsibility, and testing’s contributions are understood, you will see that quality starts to become an integral part of every discussion.
Stable adoption is the ultimate key to long lasting success, but it also takes the longest, so be patient, and know that ‘the new normal’ will take several cycles to get bedded in as the new default.
Testing transformation: Why now?
Whereas it used to take time to set up development infrastructure to build new functionality and products, new environments or frameworks can be spun up in minutes now, leveling the playing field for smaller players and teams. The cloud, more distributed architectures, and the spread of DevOps practices has granted the industry greater flexibility in speed and delivery. The convergence of all of this means that any organization that is leaving testing out of the transformation is going to get left behind.
“As a leader, my job isn’t necessarily the day to day operations, but to be 12-18 months ahead of my team so I can show them the path they need to head towards. Don’t worry about how far through the journey you are, but ask whether you’re providing value to the organization and keeping ahead of the engineering teams.”— Alex Kyriazis, Senior Manager in Testing, ANZ
Though launching a digital transformation may seem daunting, not changing anything is also a decision with consequences of its own. Transformations begin with small changes, and are best thought of as a constant process, and a daily part of your job. The journey never finishes – so keep an eye toward the future and measure your success in milestones along the way.