reduction in test maintenance effort
boost in performance test execution
Twinformatics is an information technology company that develops and operates software solutions for the Vienna Insurance Group, Wiener Städtische Versicherung AG, and Donau Versicherung AG. From design and development to operation and maintenance, they manage services and solutions, end to end, for about 25,000 employees and millions of customers. Their areas of responsibility also include the operation and development of SAP-based core insurance systems, from existing systems and claims systems through debt collection and disbursement, to group-wide financial controlling and HR solutions.
With insurance applications and services increasingly customer-directed through web applications, system performance is a critical competitive business driver. The Twinformatics performance testing team consists of one internal test manager and three external consultants (Triscon) testing four major and eight minor releases each year, plus on average one hotfix per month optionally.
In today’s ultra-competitive insurance industry, potential customers and policyholders have zero tolerance for slow or buggy applications. They expect a fast, frictionless Amazon-like end-to-end digital experience — on any device, anytime, anywhere. If they don’t get it, they’ll look elsewhere. After all, the competition is only a click away.
Markus Bonner oversees Release/Test Services at Twinformatics. The company started with testing software mainly with its internal users. There was limited risk since any performance issues were kept within the insurance company and not visible to customers.
Twinformatics then decided to open up its applications to customers and allow them to directly interact with the core systems through a web frontend. For example, they could sign up for an insurance policy.
“Performance testing is important to ensure that whatever we show our customers is really working and performing. Quality and performance is a big business issue now,” says Bonner.
Twinformatics started load testing with JMeter but found it required a lot of effort to create and maintain its test cases.
“The original plan was to run performance tests with every major release, four times a year. So we had to dedicate ~80 man-days per year just for maintenance. This was costing us something like €80,000, so it was obvious that we needed a solution that takes less maintenance effort,” says Bonner. “Then it was an easy business case to go with Tricentis NeoLoad.”
NeoLoad meets our requirements for iterative load testing for different applications and platforms with low maintenance effort.
Wolfgang Gaida, Test Manager at Twinformatics, also saw issues with JMeter and explained that once Twinformatics opened up its software to its customers, “We started seeing performance and stability issues in production when there were, say, 100 or 200 concurrent users.
Gaida knew they needed a better solution. “I had experience load testing with NeoLoad for a few years, so I knew it would meet our requirements for iterative load testing for different applications and platforms with low maintenance effort. And it’s easier to record new test scenarios [than with JMeter].”
Specifically, while JMeter has a GUI with record/playback capabilities, testers still need some specialized coding expertise to further customize scripts. NeoLoad is different because everything about test design is drag-and-drop. Users can create even large, complex tests without the hassle of coding by hand. With its frameworks approach, NeoLoad also automates the time-consuming, painful task of correlating dynamic parameters and variables.
“With NeoLoad frameworks, the ‘rules’ change the parameters automatically and correctly for each test iteration. So the [update maintenance] effort is not as great as with JMeter. We have reduced the maintenance for each test set from 20 man-days to four to five hours. We’ve gone from executing four test cases a year to about 365 — a 90X increase. And we now cover 7X more systems,” adds Gaida.
What’s particularly noteworthy about this 90X boost in testing is not the overall number of tests, but the increased risk coverage. A common problem with traditional testing approaches is that they focus on counting the number of test cases run, rather than on measuring business risk. In other words, it doesn’t matter how many tests are run if teams are not testing what’s most important.
Twinformatics has been way ahead of the curve on tackling this challenge, having adopted Tosca and its risk-based test optimization approach some 16 years ago. Tosca’s combined smart test case design and requirements prioritization helps Twinformatics align testing activities with business priorities, focus software testing resources on the tasks that matter most to the business, and stop creating, maintaining, and running tests that don’t add value.
“Before Tosca, we had — depending on who you asked — something like 10-20% coverage, without really having any real idea. Now, with Tosca and our risk-based regression testing, we have a target of 70-80% coverage, with 70-80% of this coverage automated. And now we’re doing this too for load and performance tests,” explains Bonner.
One way Twinformatics is able to get the highest possible risk coverage with minimal effort is via the native Tosca-NeoLoad integration that enables you to convert functional tests into performance tests with one click. Essentially, NeoLoad takes functional tests and “flips” them into performance tests so that you don’t need to maintain two separate sets of tests for performance testing and functional testing. Just keep your functional tests up to date, and you’ve got exactly what you need.
“We don’t need to discover new scenarios for load and performance tests separately. We can use the Tosca test cases. It’s very easy; it’s only one click to define a proxy and then we play the Tosca test cases to have performance test cases recorded in NeoLoad. And if the framework parameters change, we automatically have an updated [performance] test case,” says Gaida.
This “single definition” capability dovetails perfectly with the Twinformatics risk-based testing approach. They have developed a process that ensures tight collaboration between business stakeholders and testers. “The business sees that we have coverage of business processes according to the business risks. We use Tosca requirements for business-risk prioritization, so it’s helpful when the business can see what we do and can ‘weight’ their processes. When we first started, every process was, of course, equally important. So we got together and asked, what is the impact if something fails? How often does this process run? And with that, we came up with a business-risk rating process — which is the basis for prioritizing test cases,” says Bonner.