1. HomeArrow Icon
  2. Case StudiesArrow Icon
  3. Large-scale, multi-cloud vendor performance testing for AWS CloudFront migrationArrow Icon

Major broadcasting company

Large-scale, multi-cloud vendor performance testing for AWS CloudFront migration

K+

concurrent virtual users

default

ms

response time threshold

default

K

requests per second throughput

default

Company overview

A major broadcasting company in Poland, one of the largest multimedia groups in Central Europe, was migrating its digital content delivery and services to the cloud, specifically AWS CloudFront. The company wanted to ensure that it delivered the best quality content to its subscribers all over the world — no matter what platform or device they were using — when facing peak load traffic.

The company’s IT environment was highly complex. Although they had mature Dev and QA processes in place, the QA team was looking for a modern performance testing tool that would allow them to replicate realistic, geographically distributed, very high production traffic just before expected prime-time traffic peaks. They needed to ensure that:

  • There was no regression in performance due to the migration to AWS CloudFront.
  • Applications were optimized so that traffic spikes didn’t lead to increased cloud production costs (i.e., increased capacity acting as a bandage for buggy apps).

Furthermore, the dynamic load testing environment needed to be able to support multiple cloud vendors (AWS, Azure, Oktawave) spread across multiple zones.

Solution

The company partnered with Soflab Technology, Poland’s leading provider of testing and QA services, which recommended Tricentis NeoLoad to address the customer’s most important requirements.

With a large migration project like this one, there are a lot of complex risks to mitigate: user experience, cost, performance, etc. That’s why it’s important to compare performance baselines of what things were like before the cloud migration and what things were like after the cloud migration.

  • Arrow IconAvoid performance regression in migration to AWS CloudFront
  • Arrow IconHigh-speed, high-velocity, highly complex performance tests
  • Arrow IconNeed to use multiple cloud solutions (AWS, Azure, Oktawave) to generate load
  • Arrow IconNeed to handle a very high volume (200,000+) of virtual users

Accelerating and simplifying the migration to AWS CloudFront

As Michal Figiel, Head of Technical Testing at Soflab Technology, noted, “The migration of even partial solutions to the cloud brings with it quite a large reconstruction of the existing system. It’s not enough just to move application servers or databases to the cloud. In the majority of cases, migration to the cloud involves adapting some — or all — of the applications in such a way as to effectively gain all the benefits of the cloud solution.”

Figiel said that “practically until the day the performance test started,” dev changes were constantly introduced to the platform migrating to AWS CloudFront. “Thanks to NeoLoad, we could react very efficiently to such changes. Doing these tasks in NeoLoad was already significantly easier — and the workload much lighter — than with competing tools,” he explained. Not only was NeoLoad’s drag-and-drop, point-and-click approach to test design much faster than coding scripts “by hand,” he added, but the automated script updates were a real force multiplier. NeoLoad virtually eliminates manual script maintenance. NeoLoad updates only the part of the test that’s changed and keeps everything else from the original design the same — all the heavy lifting is done automatically behind the scenes. “We could prepare a dedicated framework to significantly accelerate the work of subsequently modifying test scripts,” Figiel said.

Building a large-scale, dynamic, multi-cloud testing environment

As Soflab’s Michal Figiel put it, “Defining and building a cost-optimized load environment for cloud performance testing is not a trivial task. It cannot be assumed that a single load generator will support 500 or 1000 virtual users. That depends on many factors, such as the complexity of test scripts, amount of data downloaded/sent by the script, traffic characteristics and policy, predicted number of errors, and so forth.”

The team adopted a hybrid approach to the load testing environment. They chose NeoLoad’s cloud platform as their primary solution, “due to the ease of management, high reliability, dynamic infrastructure capabilities, and predictable billing model,” said Figiel. But they also needed a backup environment based on three other cloud providers — AWS, Azure, and Oktawave. “We also needed to divide available cloud load generators into zones so that the flood of responses from customers’ environments didn’t overload a single load generator,” added Figiel.

With over 70 locations around the globe, hundreds of servers at each location, and no single point of failure, NeoLoad scales tests as large as the media company needed. NeoLoad is vendor-agnostic — all the major cloud providers are supported in the same platform — which enabled the company to run high-volume, high-velocity performance tests in a multi-cloud environment with real-time feedback loops. Testing infrastructure is optimized dynamically, so testing costs are predictable, kept under control, and irrelevant to development costs.

Running complex performance tests at speed and scale

Michal Figiel’s team ran dozens of large-scale performance test cycles for the company not only for the migration to CloudFront but also just before any media event that may result in a significant increase in users. They simulated traffic of web, mobile, and SmartTV platforms. Figiel said that now “with the correct configuration and availability of the cloud, the services based on it are almost impossible to kill.”

He went on to explain that in migrating to the cloud there are many pitfalls that may be identified only by simulating real-life traffic. Large-scale test cycles uncovered, for example, common issues like VM provisioning thresholds and VMs getting up and running too late for an aggressive traffic increase (e.g., during prime time or a breaking news broadcast).

“Although in theory cloud solutions allow for provisioning ‘unlimited’ bandwidth or computing power, in practice a number of requests, exclusions of certain protection mechanisms, or obtaining appropriate permissions are required in order to properly perform performance tests on a large scale,” Figiel continued. “This is why we used NeoLoad’s cloud platform for testing purposes — all reports, protection exclusions, or permissions are covered. From the point of view of executing tests, it comes down to just booking and running a test session. And the NeoLoad cloud controller and load generators are preconfigured to handle much more load than a single rented machine in any cloud solution. This makes it incredibly easy to build and maintain large load environments.”

More than 20 test iterations were performed as part of the test cycle. After each iteration, the team was able to pinpoint any problems (errors, slow system performance, or unavailability) and implement configuration or platform code changes. In the final iteration, traffic significantly exceeded the assumed 200,000 concurrent users, with the transfer rate exceeding 90Gbps and throughput over 200,000 req/s, while maintaining full usability of the client portal and request response times below 30 ms.

“In the end — apart from technical and organizational expertise, and NeoLoad being the best tool on the market — success came from the tight cooperation among people: the client’s IT team, Soflab Technology, and an outstandingly supportive Tricentis team,” Figiel concluded. “It’s also worth pointing out that thanks to NeoLoad, precise preparations, and tight coordination, all of the test iterations and configuration fixes were completed during just 5 nightly hours of testing.”