How to scale your load tests with NeoLoad

You’ve got complex load tests coming up with a deadline. Your virtual user (VU) load is significantly higher than anything you’ve tested before. As you prepare your scripts, you’re probably also thinking about load generator requirements — differentiation between them, their capacity, and how many needed.

No manager or company wants to suffer the consequences of wasted time and budget as a result of failed testing due to unused or missing resources. The following is a guide to help you understand and discover scaling your resources to meet load testing demand using Tricentis NeoLoad.

Load generator capacity planning

Once the capacity for a single load generator is known, you can then extrapolate the number of total load generators (LG) needed to meet load test demand. For example, if the internal machines you’re using for LGs have 1,000 virtual user capacity based on current scripts and configuration, and you need to run a 10,000 VU load test, you will need 10 LG machines. Ultimately, addressing what each of your LGs is capable of handling is the key.

When figuring out LG capacity and performance, you should keep some core factors in mind:

• Physical memory
• Allocated heap space to the Java virtual machine (JVM)
• CPU capability and bandwidth (local hardware, and network between)
• Storage location, complexity of design, protocols used
• Kerberos authentication (massive memory overhead), and XPATH extractions

Having a quad-core processor (or better), rather than dual-core, is a must for heavy load testing today. Having 12-16 GB of physical RAM, and allocating one-fourth to one-half heap space for the JVM is recommended. Having gigabit network cards (NIC) for bandwidth will alleviate LG communication and traffic bottlenecks. It’s critical to note that script design can have an adverse impact on LG load capacity. Refrain from using excessive response validations, avoid Kerberos authentication if possible (can stunt load capacity as it’s known to max out at 50 VU/LG), and watch out for redundant loops in transactions.

Load generator capability

Once you know the hardware or cloud services you plan to use for load testing, you need to determine the capability of a single LG with your application and scripts. Best practice approach would be to create a load test using one LG/script(s) combination involved, configuring it to ramp up at a specific rate. Then, watch your load tests and monitor how long it takes for the LG to run its load. Make a note of virtual user load at that time, and remember its limit. It is good practice to take 5-10% off of this threshold to account for a marginal safe zone (to configure the rest of your LGs’ capacity). It is not wise to run your LGs at maximum load, hovering at the maximum limits. A load of 80-85% of any specific resource limit in your test is ideal. For example, if your load ramps up to 700 VU while your memory is at 100% utilization, review what your load was between 80-85% memory utilization. It could be in the ballpark of 575-600 VUs, thus establishing your planning point. If 600 VU is your mark that puts your LG at 85% of memory utilization, you can now safely and confidently identify total LG count for your full load. In this example, if your load totaled 5,000 VU, you would plan for a minimum of 9 LGs (5,000 VU / 600 VU per LG = 8.33, or 9). Consider the following ramp-up test configuration example:

As part of the setup, you would start with a set number of virtual users (this case, 100 used), adding the number of users over time or completed iterations. The above also displays the addition of 10 virtual users every 30 seconds. Your particular scenario might necessitate a more aggressive ramp-up (e.g., 400 VUs initially, with perhaps up to 50 VUs introduced every 30 seconds). It depends on where you think the load will hit peak capacity, causing a bottleneck, and what kind of hardware will backbone your LGs.

Resource utilization

After running load tests that ramped up the load over time, you’re going to see resource usage. See the following high-usage example due to memory:

You can tell in the above that as the memory usage nears 85% (eventually near 90% and above), the load is at 155 VUs — the limit for this LG (based on existing scripts and configuration). Use this to calculate machine total. In this case, it would take 10 LG machines to run a load test for 1,500 virtual users. Note that an LG may have an archetypal capacity for 250-5,000 VUs per machine (in some cases even more), depending on many factors.

There are other options which could increase the LG capacity. More significantly, you could improve overall system memory or heap space memory allocated to the Java virtual machine. In this scenario and configuration, perhaps only one-fourth of the system memory of 8 GB (default setting) might be assigned, setting it at 2 GB of heap space for the Java process running the load on this machine. (Oracle JVM is limited to 1.5 GB of heap space if using the 32-bit version.) By doubling this memory allocation, you could see a similar or greater lift in machine capacity, as objects in memory are shared and re-used. Another significant consideration is to make sure you have the maximum processing power available.

There is also an inherent danger of using too much heap space allocation for the JVM — more doesn’t always mean better. Having greater system memory than what is needed can potentially degrade performance. The Java heap is where the objects of a Java program live — e.g., Java process for the LG. It is a repository for live and dead objects, free memory. When an object can no longer be obtained from any pointer in the Java process, it’s marked as “garbage” and ready for collection. The JVM heap size determines how often/long the VM spends collecting garbage. If you set a large heap size, full garbage collection is slower but occurs less frequently. If you set your heap size by your memory needs, full garbage collection is faster and more frequent. The heap size tuning goal is to minimize JVM garbage collection while maximizing concurrent virtual user volume during the load test on each LG. An error indication with heap space memory would have the following log association java.lang.OutOfMemoryError.

Conclusion

To resource plan, you first need to determine load generator capacity. Remember to run a ramp-up test on one of the LG machines to establish its maximum size; scale back to 80-85% of the max load, using this mark to calculate your total number of LGs required for your load testing. Pay attention to the factors that affect LG capacity. Have your desired outcomes in mind from the beginning, putting good design practice into scripting and resource configuration. Keep your designs as simple as possible. Plan for your resource needs, including hardware. Configure these resources to maximize their support capacity and delivery.

The post was originally published in 2018 and was most recently updated in July 2021.