Large media company

Performance testing a complex video streaming environment at speed and scale

Company overview

A large Australian media company held exclusive rights to broadcast the final season of one of the most popular TV shows of all time, live and on demand, over their digital IP-based video streaming platform. The company needed to ensure streaming quality during spikes in viewership — expected to be on the order of 2–3X the volume of traffic of anything the company had seen before — especially for the highly anticipated series finale.

While such high-visibility programming offered high reward for the media company, it also posed high risk. For instance, another media company had recently experienced a failure during a major sporting event (which had been streaming to a smaller audience than the TV series), and the fallout included widespread bad press, lost subscribers, and being forbidden to bid for broadcast rights for the event in the future.

Six weeks before the series premiere, the media company turned to AccessHQ, a leading Australian IT quality assurance and testing organization, to de-risk the potential for capacity overloads during peak traffic during live and subsequent video-on-demand streaming.

When AccessHQ was called in, they faced three challenging pressures: scale, speed, and complexity. They had only six weeks to develop an effective strategy and execute one of the largest load tests ever run in Australia. They needed to test not only the company’s core video delivery platform but also many specialized third-party services that are tightly integrated into the end-to-end customer journey, from sign-up to navigation to licensing. AccessHQ would need to provide actionable feedback to both the media company and third-party vendors in time to optimize system performance before the season premiere.

Challenges

“It was basically a best-endeavors de-risk for [the media company],” says Reuben Korngold, Consulting Director at AccessHQ. “With only six weeks, we didn’t have the time to test the way we would’ve liked, but we could reduce the risk from a high level to a more confident level. Everybody said it wasn’t possible to test HD video streaming at this scale, but the [media company’s] COO said, ‘I want to make sure that if half a million people get on and stream, they’lI be able to watch HD.’”

There were concerns that the media company’s existing load-generation tool would not be able to handle the volume, and whether other performance testing solutions would be able to provide the level of scalable cloud capacity. This was a crucial factor, as AccessHQ knew they had to push the system to its limits, simulating up to triple the number of users streaming HD video beyond what was known to be the media company’s previous peak. They could not tether themselves to any single cloud provider. Given the size and scope of the test, they needed a vendor-agnostic, multi-cloud testing solution that would allow them to run load tests across multiple data centers from a combination of Amazon AWS, Google Cloud, and Microsoft Azure.

Key obstacles included:

  • Simulate HD streaming video at hyperscale (1 million+ viewers)
  • Viewership expected to be 2-3X greater than anything the company had ever seen before
  • High-visibility, high-reward/high-risk programming event
  • Only a 6-week performance testing window, with inflexible schedule
  • Highly complex system of myriad third-party integrations
  • Needed vendor-agnostic, multi-cloud capability
  • Arrow IconA large media company ran the biggest load test in Australian history to ensure flawless video performance of a TV series finale with unprecedented traffic.

“Tricentis NeoLoad’s scalable cloud capacity was key. We used more than 1,900 load generators across AWS, Azure, and Google Cloud.”

Reuben Korngold, Consulting Director, AccessHQ

Solution

AccessHQ partnered with Tricentis to leverage NeoLoad’s cloud platform to simulate the customer journey end to end for hundreds of thousands of concurrent users.

Says Reuben Korngold, “NeoLoad’s scalable cloud capacity was key. We used more than 1,900 load generators across AWS, Azure, and Google Cloud.” NeoLoad also natively interacts with container orchestrators such as Kubernetes, OpenShift, Microsoft AKS, Amazon EKS, and Google GKE to provision and use load testing infrastructure automatically, on demand.

“NeoLoad was the only product that we could identify with a specific video player component to drive the load testing. Some of our testers had been looking at how they could try to simulate themselves, and the fact that NeoLoad had this capability meant that we would already be getting video player metrics,” says Korngold. “The fact that we’d done another video streaming test previously in NeoLoad, albeit smaller — just some movie trailers for one of the cinema chains in Australia — also gave us confidence. We’d actually scripted and executed performance tests using the video capability before.”

“We were pushing boundaries,” Reuben explains. “We had that faith from our previous dealings with NeoLoad that these were the guys to work with who could help us make it happen. I’d just get on a call and say, ‘This is what we’re thinking — can you support us?’ If you tried that with other performance tool vendors, you’d get a level-1 help desk person. Instead, I’m picking up the phone and getting connected with the right people. We’d all get on a call and solve some very complicated issues in rapid time. We had only a six-week project schedule — it could take you that long to get a hold of the right person at another performance testing vendor. You just wouldn’t even want to try.”

  • Arrow Icon“We were pushing boundaries. Tricentis are the guys who could help us make it happen.” — Reuben Korngold, Consulting Director, AccessHQ

Identifying third-party bottlenecks in a complex end-to-end flow

AccessHQ ran the load test four or five times. “If we hadn’t done that testing, there’s no way [the media company] would have succeeded. And as is always the case, we found that it wasn’t just their core delivery network that needed to be optimized, but also many third-party services.”

AccessHQ had to dig into the highly complex end-to-end flow of all the different components and systems, testing 550K requests per second. Streaming video involves a lot of moving parts. Reuben Korngold explains: “For instance, when you press PLAY, it goes and acquires a digital license for rights to play the video. Until those rights are issued, you can’t actually watch the video stream legally. There’s a process flow that went out to a third party in the US to issue this license in order for it to come back. When we did the load test it became apparent that this was a bottleneck. So there were a lot of third-party bottlenecks that got unblocked and identified as things to be optimized or to be scaled in preparation for that event.”

In the end, the media company averaged 681K viewers for the season, with over 1.4 million people streaming the series finale. It was the largest single-day video player measurement (VPM) rating in Australian history — and it all went off without a glitch.

 

Key results: 

  • HD streaming video quality sustained for over 1.4 million viewers
  • One of the largest load tests ever conducted in Australia
  • Tested 550,000 requests per second end-to-end across complex system
  • De-risked performance problems within inflexible, 6-week schedule
  • Able to utilize 1,900 load generators across AWS, Azure, Google Cloud

 

  • Arrow IconWe had only six weeks to execute one of the largest load tests ever run in Australia.” — Reuben Korngold, Consulting Director, AccessHQ
  • Arrow Icon