Case Study

Office Depot

Office Depot improves load testing productivity
66%
reduction in scripting and test maintenance time
image

Office Depot matures SAP Hybris load testing while reducing testing effort 66%

Evolving from JMeter load testing

Office Depot Europe is currently migrating all EU legacy websites onto a single SAP Hybris web platform. The Hybris web application has a three-tier architecture. The production and performance test environments are hosted externally to Office Depot Europe within the Hybris cloud.

For the initial pilot rollout, it was decided that JMeter would be used for load testing. After the successful pilot go-live, the QA team was tasked with carrying out a technical proof of concept to identify the preferred load testing tool for future performance testing. They needed a tool that met technical criteria defined by the QA team and also addressed some of the drawbacks they encountered with JMeter. For example:

  • Limited reporting functionality built into the tool
  • Stability problems under load and extended durations of time
  • Complex interface did not allow for cross-training throughout the team
  • Limited ability to integrate Selenium to measure front-end performance
  • No possibility for client-side mobile testing within JMeter

NeoLoad helps Office Depot improve their load testing by speeding up in-sprint testing.

NeoLoad beat the competition

After evaluating LoadRunner and StormRunner, Office Depot decided to go with NeoLoad for their load testing needs because:

  • NeoLoad satisfied the majority of the technical criteria identified before the proof of concept (e.g., ease of use for script development and scenario development, built-in analysis and reporting, available hosting platforms)
  • Ease of use for cross-training throughout the QA team
  • Customizable reporting to share with key stakeholders
  • Integration with other toolsets used within the team, such as Selenium and Jenkins

The teams work in an agile environment, with several performance tests executed against each sprint. The QA team also uses Selenium for test automation and Jenkins for CI. They are now in the process of integrating with NeoLoad for front-end performance statistics and more frequent performance testing against builds during each sprint.

Next steps with NeoLoad:

  • Continuing regular in-sprint testing
  • Integrating Jenkins and NeoLoad for automated in-sprint testing
  • Integrating Selenium and NeoLoad to measure front-end performance
  • Cross-training all QA team members in performance testing

“Our teams work in an agile environment, with a number of performance tests executed against each sprint […] NeoLoad has improved our load testing with faster in-sprint testing .”

Mikey Warner — Head of QA

The benefits of using NeoLoad for SAP Hybris testing

So far, NeoLoad has helped improve their load testing in the following ways:

  • Faster execution of tests within each sprint
  • Improved reporting to stakeholders
  • Integrating Selenium scripts within NeoLoad tests quantifies front-end performance alongside the back-end performance statistics from NeoLoad
  • Additional members of the team are cross-trained and running tests in NeoLoad to allow earlier scripting within the sprint

NeoLoad has added significant business value to Office Depot with respect to:

  • Improved application stability. For example, it helps with tuning JVM garbage collection settings and reduces the need for additional infrastructure in production
  • Improved testing productivity by reducing scripting time and test maintenance time by 66%
  • Earlier identification of performance issues, resulting in improved customer satisfaction
  • Ability to cross-train and upskill other team members