Why (and How) Toyota Adopted an Agile, Cloud-Based Approach to Load Testing

What do Priuses have to do with agile, cloud-based load testing? Both are evidence of Toyota’s commitment to innovation and modernization.

Toyota Motors North America shared their journey to Cloud Load Testing (with Tricentis Flood and AWS) at Accelerate SF 2019. This keynote room session, delivered by Neeraj Tripathi and Hector Martinez from Toyota, was one of the most well attended and top-reviewed sessions of the conference.

As either a tease or a recap, here are some of the key points…

Why did Toyota shift to Cloud Load Testing?

Toyota has a commitment to innovation that extends beyond its cars to all of the ways in which they deliver their products. This includes a desire to maintain a lead in terms of development best practices, and a drive for constant improvement in the way software is being created. The main factors that drove Toyota to replace its legacy load testing tools with a more modern, cloud-based approach included:

• Desire to shift testing left: More and more organizations are moving towards shift left load testing, with the demand that more load testing be done earlier in a development cycle to catch performance regressions while they are still cheaper and easier to fix.
• Complex maintenance of on-premise resources: Managing on-premise load generators was becoming a costly and time-intensive exercise. Additionally, being limited only to running tests on the physical servers that you own can limit the amount of parallel load testing which can be done, which can be a major bottleneck when moving to an agile and DevOps environment.
• Previous performance testing tool was not flexible enough to meet the needs of agile and DevOps: the closed source, proprietary scripting model was not being adopted well by developers, who were not able to contribute to scripts to load test their new features being added in each sprint.
• No solution for early testing: testing was always happening late in the development cycle, often leaving very little time to get these issues fixed before needing to push code into production.

What were Toyota’s key criteria in a Cloud Load Testing offering?

Toyota had a number of options when evaluating potential solutions for their Cloud Load Testing platform of the future. When assessing the various tools on the market, their selection boiled down to a few key criteria:

• Scripting: Ability to support common open source tools like JMeter, with a tight native integration to this tool.
• Management: Ability to control the various teams, projects, cloud resources, and integrations required when testing across a wide array of teams in an enterprise like Toyota.
• DevOps Integration: A well-documented API for integrating load tests to the CI/CD pipeline for continuous load testing.
• Execution: Ability to execute load in your own AWS or Azure account, to take advantage of the ability to use VPC’s and other security controls to test behind the firewall environments.
• Reporting: Ability to easily report on test executions, compare historical results, and drill down into any major issues quickly.
• Customization: Ability to extend and customize the tool to fit into the particular CI/CD ecosystem at Toyota.

At the end of the day, Toyota Motors North America evaluated many potential Cloud Load Testing solutions, but ended up choosing Flood for having the highest marks across their key criteria.

What benefits has Toyota seen with Cloud Load Testing so far?

Since implementing this new approach to load testing, Toyota has seen tremendous benefits in their organization’s software delivery process. Specifically, the main benefits of adopting this new approach include:

• Quick Adoption: roughly 40 separate teams have already adopted this cloud-based load testing solution, which has been named the Agile Load Test Platform (ALTP).
• Frequent Execution: teams are testing a combined 700 hours per month already, with some of the longest tests running up to 24 hours.
• Integration to the CI/CD pipeline: by using the Flood API, teams have been able to integrate closely to their CI/CD pipelines and trigger load tests with each and every build.
• Security Team approved: through using their own AWS account, they can securely access environments that are both publicly and privately hosted, with the blessing of corporate security.

We look forward to seeing Toyota take this solution even further in the future, as they expect to ramp up almost 100 teams on this new approach. With the addition of our new Teams Management feature in Flood, they will be well positioned to control and administer the platform to this massive population of new users.

How can I build my own Cloud Load Testing platform?

Teams interested to experiment with building their own solution like Toyota’s can access the downloads and trials of the various required components here:

• JMeter Download
• AWS Signup
• Flood Trial

If you have any questions about how you can get started with cloud load testing, please don’t hesitate to reach out to us – we’re always happy to help you get started.