In the world of retail banking, consumers expect more than just checking accounts and debit cards – they expect the right set of solutions to transform their financial lives. In 2004, Q2 founder Hank Seale set out to build stronger and more diverse communities by strengthening their financial institutions through technology. To achieve this, he developed a single platform that removes technical barriers by integrating with legacy financial institution technology. Today, one in every 10 digital banking customers in America are signing into Q2’s single platform to bank through their chosen financial institution.
Q2 has reinvested over 20% of their annual revenue (approximately $500 million in 2021) on R&D and built out an industry leading in-house usability testing facility focused on delivering the best possible experience. Q2’s product philosophy – “Ask. Understand. Deliver.” – means software updates must be delivered at the speed of customer anticipation and with the utmost quality. In order to meet that goal, Q2 releases new versions of its digital banking software multiple times per month for web, iOS, and Android.
“We are in the fintech industry – testing is critical,” says Jan Acosta, Vice President, Digital Banking. “We must have well documented tests that show very clear traceability on what we executed. In order to achieve the coverage and quality we need, a large emphasis is put on automation – we can’t do it all by hand and release at the cadence we want to.”
The team at Q2 has been moving to greater levels of automation. They leverage multiple automation tools and frameworks to ensure quality for each release, including Selenium, Appium, and homegrown frameworks based in Python. With so many tools and processes across teams, it was often difficult to obtain a clear picture of where a given release stood. The team needed a way to holistically trace issues back to specific tests and environments, and to communicate those issues back to the development teams. Leaders relied heavily on gut feelings about where the release readiness stood instead of being driven by data.
While some teams had access to various types of test case management tools, high levels of automation across a disparate, unintegrated toolset created bottlenecks and visibility issues. To address these issues and continue to scale their DevOps progress, leadership at Q2 determined a clear need for a more streamlined approach to test management.
“Test effectiveness has to come first,” Acosta says. “It is born of test strategy and test planning and having a good relationship with product and development, so you can understand what you are being given. That allows the testing team to tease apart the code and understand what the permeations are that can be checked for.”
This philosophy helped to guide the team as they built a new test management strategy and searched for a tool that would streamline their DevOps pipelines and push innovation forward.
After evaluating the organization’s existing tools and what was available in the marketplace, the team selected Tricentis qTest.
qTest’s ability to integrate with tools across the DevOps pipeline was key to the decision. Acosta tells us, “The richness of the qTest API is one of our most critical selling points. We are able to take a tool and make it work for our world – it was malleable to where it could fit into our process without having to change how we do things.”
Organization of projects, processes, and teams was an important first step. Projects in qTest are broken up by functionality and integrated with specific boards in Jira. Because of functionality overlap from one team to the next, test case sharing in qTest allows others to quickly locate and execute existing test cases – both manual and automated. This process reduces redundancy in test cases being leveraged by the multiple teams from one release to the next and makes test case maintenance across teams easier to facilitate.
“The biggest factor was setting standards,” says Acosta. “Being organized when you are dealing with a large team and several large projects helps create more effective management of resources.”
A top priority was implementing specific guardrails to ensure consistency and ease of collaboration across teams. One example was the naming convention for test assets. Because these fields are customizable in qTest, Q2’s test cases now follow a specific naming convention that matches Bitbucket and other repositories, so both testers and developers can easily identify what they need when test cases need to be addressed.
Custom fields and statuses have also helped to evolve the testing process for Q2. For example, in addition to custom fields indicating where the test is run (mobile, web or both), custom statuses are leveraged in order to triage failed test cases. Failed test cases trigger a review to determine the source of the failure.
The team realized failures were sometimes due to server downtime, mobile devices being offline, network issues, or other test environment issues not directly related to defects in the code. This visibility has allowed the testing team to work more closely with various departments within IT to optimize how and when they are executing test cases. This move has significantly reduced false positives caused by environment issues.
“qTest has given much greater confidence with the larger development organization and even more so with the delivery and support organization. I can now SHOW them what I did. It has allowed us to make data-driven decisions instead of gut feel decisions. Data-driven decisions take the emotion out of the conversation to determine how to best proceed forward.”
Integration across the DevOps pipeline has streamlined work across the Q2 team. When a developer checks in code to Bitbucket, a job is triggered to test environments for the new build. Once the health check is completed, a Jenkins job is triggered for automation, and all test results are logged in qTest. To keep other stakeholders up-to-date, a workflow triggered by qTest Pulse shares results to a Microsoft Teams channel and sends an email notification to the team.
This process allows for greater confidence in each release. The Q2 team now has a clear understanding of where the release stands and what’s required before release.
“This has been an evolution from having gut feel reactions to having tangible metrics,” Acostasays. “If I go back to the beginning before qTest, we would do a daily standup in prep for releases. I would hear statements like ’we are on track’ or ’we are fine.’ This did not tell me what percent complete we are toward the go live date. Today, I get ‘we are 93% complete’ or ’we are blocked by issue #blank in Jira.’ This visibility into what we are doing creates so much more clarity and confidence for the broader organization.”
Data is at the center of every decision today, both for release readiness and in determining ways the team can continue to improve their DevOps process. Stakeholders from every level of the business rely on this data to paint a clear picture of where a release stands and measure progress towards goals – including dev managers, senior leadership, scrum teams, and test strategists. Q2 leverages qTest APIs to pull information into the data warehouse every 24 hours. PowerBI is used to build custom reports and dashboards that are critical for decision-making.
Data from each release is leveraged to manage both execution and test metrics. Execution metrics, such as what percentage of test cases passed, how many were based on mobile testing, what percentage of defects were reported in Jira, are used to assess the readiness and quality of each release. Test metrics are used to determine how and where the teams can improve. Examples include: what percent automated are each of my teams (ensuring automation is not declining as functionality and test assets grow), what features need to increase automation, or where are we finding the most issues.
The ability to pull critical data from a central location across teams with qTest has helped facilitate this process and allowed for the continued growth of Q2’s innovation and quality.