testing authoring time, reduced from 3 days
reduction in troubleshooting time
Our client is the market-leading data authority for the hybrid cloud. It provides a full range of hybrid cloud data services that simplify application management and data across cloud and on-site environments to accelerate digital transformation. Its solution empowers global organizations to unleash the full potential of their data to expand customer engagement, foster greater innovation, and optimize their operations.
The client’s application was complex and included many user flows, requiring the creation of thousands of functional tests with the goal of shifting left. In addition, they needed to test as close to development as possible.
Selenium was their first choice and a team of over a dozen testers was assembled to begin the task. After a few months, it became apparent that it was going to take more time and manpower to achieve their goal of CI/CD. The tests were complex and took three days to author. To make things worse, the tests would often break, leading the team to spend extra time maintaining and fixing the tests. Shifting left would require teaching the developers Selenium as well.
The lack of tools to troubleshoot a failed test (e.g., screenshots to compare, detailed error messages pointing to the right step, test history, or the parameters over the flows) led to long time-to-resolution involving a number of team members. A lot of time was wasted, not only trying to figure out why the test failed but to also explain the discoveries to the developers.
“Maintaining tests took a lot of my time. When developers run tests that fail it becomes more of a distraction than confidence in bug prevention. Both groups had to stop what they were doing to figure out if its the functionality or if it’s the test. We found ourselves spending more time trying to stabilize the tests than actually testing.” — Company QA Manager
“Today we have hundreds of tests running on every pull, giving the developers feedback to their code within minutes. The developers themselves easily update test scenarios so QA can focus on increasing coverage. We also significantly reduced our cost of quality: The rich information we get allows us to reduce the time to troubleshoot by 80%.”
As a rapidly growing enterprise, they needed a way to optimize processes through automation. This plays a significant role in their move to Agile development. Within a couple of months of using Testim, the team was able to create hundreds of their UI test scenarios. Today, they are proud to say that they are authoring tests in under an hour compared to the three days it was originally taking.
“Before Testim it would take 3 days to author a single test in Selenium, now even for the less experienced tester, writing tests takes under an hour, developers can update tests on the fly and figure out where the tests are failing without any additional help.”
However, the biggest impact was the reduction on maintenance. Tests are stable and trustworthy so when a test fails the user knows it is either due to a bug or the test requires a change in the flow. The team focused most of its time on increasing coverage knowing that the tests adapt to UI changes by the development team.
Today, we are proud to say that they are fully CI/CD, testing on every code push, and running thousands of tests every day.