regression test automation for existing applications
reduction in testing time
For 160 years, the Sydney Trains division of Transport for NSW has provided essential transport services to the residents of Sydney, Australia. More than just a way to get to and from work, their initiatives focus on creating a safe, efficient, cost-effective, and enjoyable service that meets the growing levels of freight and passenger demand – which is expected to reach 40,000 customers per hour by 2024.
“Our goal is to serve the public and to move them safely on their full journey – spanning their service on roadways, waterways, and trains,” says Ernie Lam, Senior Manager of Digital Products and Applications at Sydney Trains. “We look at transport as a whole, not just point to point. We want to get you all the way from your home to wherever you want to go and ensure your travels are as seamless as possible in an exceptionally large area.”
Achieving such a goal is dependent on both prebuilt and custom internal systems. These applications are designed to ensure a seamless travel experience for travelers, deploy management and safety tools for the million trips taken per day in Sydney, and facilitate the daily journeys of thousands of individuals. In addition to maintaining existing software in the tech stack, the development and testing teams work closely together to deploy new applications and feature sets throughout the year. Many of these applications are designed to run on 10,000+ mobile iOS devices, including iPhones and iPads, as part of Transport for NSW’s cloud-first strategy.
Pre-2018, software testing for these applications was completed manually by the project team, but a growing number of applications and releases required the team to find a more scalable way to test. These domain experts were tasked with not only ensuring the quality of the code being released, but also understanding usability for those using the applications, both employees and patrons, to ensure the best possible experience. Specific feature sets planned for critical applications were not only needed, but sometimes contractually obligated during specific release periods. This meant speed of delivery was more important than ever as the team planned out future sprints. “It was not sustainable to keep doing manual testing. The major focus was to build quality from the beginning and all the way through each code release. To achieve this at speed, we needed to embed test automation to facilitate that quality,” Ernie says.
When it came time to determine the tool to push testing forward faster, leadership allowed the project testing team to evaluate the marketplace for the best solution for their specific needs. An important criterion for the proof of concept was the ability to learn the tool quickly. Tricentis Tosca prevailed as an early contender because its codeless test automation can be learned quickly without scripting knowledge.
“If it had taken a year to learn it, it wouldn’t have been worth the time and effort,” Ernie says. Members of the team downloaded a free version of the tool and leveraged online Tricentis Academy programs to learn how to create automated tests. The team evaluated Tosca for both web and mobile testing, as well as its ability to scale as testing needs increased. After some quick success with the trial, Tosca was chosen as the automation tool of choice.
Since then, 20+ domain experts have learned to automate by leveraging Tosca within the IT division. As a standard, some teams even require BAs, developers, and testers to complete Tricentis’ online training, a course which equips beginners with the knowledge required to quickly begin automating tests.
“Because we believe in Agile principles continuous learning is just as important as continuous integration and continuous development. If you don’t invest in yourself, you will get left behind,” says Ernie.
When Sydney Trains started on their automation journey, they already had the foresight to consider how cloud applications would impact the development and testing of future applications. In the early 2010s, the team was already starting to move away from on-premise solutions in data centers to a cloud-first mentality with AWS. Since Tosca would play an important role in the future of these applications, the team worked to design an ecosystem for the traditional SQL database to be available for cloud deployments with physical iOS devices.
In addition to maintaining existing system upgrades, the team is consistently working to provide testing for new cloud applications.
For example, the team is currently working on the rollout of a critical internal self-service portal for train crews – allowing them to perform multiple work functions in a single location that would have previously been coordinated through a manual and paper process. This new portal will save time for employees as they update work schedules, view and manage rosters, access safety critical documentation, and complete other required daily tasks so they can focus on delivering value to passengers, while facilitating the move to digitally enabling crew members.
Today, developers can execute automated test suites within the CI/CD pipeline every time code is checked in. Executing the Tosca test suite is part of the definition of done for every piece of code. Should any of the tests fail, the developer can quickly pinpoint and address any defects in the code – which increases both the speed of testing and the quality of releases.
Testing is then passed off to the project domain experts to execute full regression suites and generate new tests for new feature sets. Automation efforts are prioritized according to risk-based assessments combined with time-based regression requirements. This allows the team to ensure the most critical and time-consuming tests are automated for future testing. This becomes especially important as new feature sets are rolled out to ensure previous functionality is not broken when new code is released. Tosca has also helped the team provide faster testing for critical applications under maintenance. One example is SafeComm, a safety critical communication (SCC) web application designed to support the completion of audio communications technical and behavior compliance with the standards set forth by Sydney Trains. The system requires domain experts to fill out a form with a considerable number of answer combinations, all of which must be tested. Previously, it took 90+ minutes to test the UI before each release. After automating testing with Tosca, the process can be completed in a third of the time.