Agile testing

How Dutch Railways Introduced a Framework for Effective Agile Test Management

Today’s testers carry enormous responsibility. At a time when software impacts everything we do, a software failure can severely weaken or even break a company. That responsibility is not lost on the testing team at Dutch Railways, which operates the busiest train network in the world with routes through the Netherlands, Germany and the UK. In the following blog, Niels Rikze, Test Manager at Dutch Railways, shares how he successfully implemented qTest for agile test management across teams. 

Making Testing a Priority at Dutch Railways

Dutch Railways has a very complex operation, managing over one million journeys each day, all with the goals of timeliness and cleanliness. It falls on the company’s testing team to not only help manage this complexity but also to ensure that its technology runs smoothly to avoid negative media exposure.

We want to make the train experience as good as possible. But you can do well 90% of the time, and the 10% that’s not good will land you on the front page or the 8 o’clock news. We want to prevent that, so that’s one of our testing team’s most important risks,” explains Niels Rikze, Test Manager at Dutch Railways.

To help mitigate this risk, Dutch Railways has a Test Competency Center with over 170 testers who handle everything from test automation to security testing. Additionally, the company invests heavily in its testers, with a curriculum for ongoing education.

Identifying the Need for Test Management

One of the biggest efforts the Dutch Railways Test Competency Center works on is the company’s mission critical application, which handles planning for all train traffic in the Netherlands and includes about 800,000 lines of Java code.

Recently, the team found they were spending an outsized amount of time reporting on testing, administering test cases and keeping results up to date — all of which they handled in spreadsheets. Recognizing an opportunity to streamline those activities, Dutch Railways began the search for a test management solution.

According to Rikze, Dutch Railways had two main requirements for the new test management solution: To support testing in smaller teams and to provide continuous insight into the quality of the systems and the risks covered during testing. Rikze and his team evaluated several solutions based on these requirements and ultimately landed on qTest Manager.

In addition to our functional requirements, one very important aspect for us was support, including the company’s willingness to help and speed in responding to our questions. That was something critical that we took into consideration when we selected qTest,” Rikze shares.

Implementing qTest Manager at Dutch Railways

Once Dutch Railways made the decision to implement qTest Manager as its test management solution, the responsibility fell on Rikze to handle that implementation. However, he encountered three core challenges at the start:

  1. Dutch Railways had 11 teams, each working in JIRA in its own way. That meant an epic, a story, a task and a defect could all mean something completely different for each team.
  2. Testers relied on extensive use cases for testing, some of which were over 130 pages long. As a result, basing test coverage on those use cases wasn’t always helpful, because if a test failed it could be due to something trivial within those 130 pages or something very critical.
  3. Finally, all of Dutch Railways’ code rolls up into one big release, meaning the team cannot re-test small elements along the way. Rather, they must wait until the build is complete to do end-to-end testing. This setup makes it difficult to inform stakeholders about product quality throughout each sprint.

Based on these challenges, Rikze sought help from Tricentis to design an effective solution. This collaboration led to a test management solution designed around the core principles of including clear acceptance criteria for each user story to provide clarity around base coverage requirements, supporting Agile teams for efficiency and providing access to instant and continuous reporting.

Putting qTest Manager to Work

Based on the solution designed with the Tricentis team, Dutch Railways has put qTest Manager to work to resolve its test management challenges.

First, the team uses a Jenkins build a pipeline to determine the success or failure of its various automated test cases.

Next, they use qTest Manager as a base to house all of their tests. Within the solution, they have three types of projects:

  1. New Business Value: These projects add net-new features to the application. Now, when testers run a test case, they use qTest Manager to link to the acceptance criteria for easy reporting. Best of all, this information comes in automatically thanks to an integration with JIRA.
  2. End-to-End Acceptance: These projects conduct end-to-end testing at the release level, including chain testing to ensure the application functions properly as it communicates with other systems.
  3. Existing Business Value: These projects involve checking that existing features still meet all of their acceptance criteria as new features get added to the application. These projects are release-driven and still rely on Dutch Railways’ legacy repository.

Finally, Rikze and his team use qTest Insights to monitor progress and determine if the team will meet their deadline. “In the end, I want to be able to ask: Are we still on track to go live on our deadline? And I want to be able to answer that question as quickly as possible. We use qTest Insights to solve that complex problem,” Rikze explains.

Continuing to Evolve Test Management

Implementing qTest Manager has already made an enormous impact on Dutch Railways’ testing operations and the team has adopted the new solution quickly.

Our testers use qTest heavily for our new business value and end-to-end acceptance projects and they’re really happy about the new solution,” Rikze shares, adding that the initial testers to start using qTest have also been big advocates for getting others to use the solution.

While Dutch Railways has already made significant progress, Rikze says this is only the beginning. Going forward, he plans to work closely with the Tricentis team to tighten reporting within qTest Insights to get even faster at answering the question of can we go live on time? and to get clear reports in place for key stakeholders.