If your development team has moved from a legacy ALM to Jira, you’re probably looking for a Jira-compatible test management tool so that you can stay aligned and ensure you’re modernizing testing process in step with development. If you’re a manual shop, or you’re only running a small number of test cases, a Jira app might be a great solution for you.
But if you’re part of a larger organization with a significant focus on testing, or if you’re planning to increase levels of test automation, Jira test management apps are likely to cause significant performance issues, specifically with Jira Server and Jira Data Center.
Jira test management performance issues
Because Jira apps lack built-in storage, they rely on the Jira database to store the data they manage or create. As this data accumulates and the app consumes more space, the entire Jira instance becomes increasingly vulnerable to performance degradation, stalls and crashes, which impacts every Jira user throughout the organization in as little as 14 weeks, according to our research.
When a test management app causes Jira to crash, testers can no longer store, access or reuse test cases, losing all visibility and traceability, which are often the main reasons they implemented test management in the first place.
To address the scalability issue, you could pay for more Jira Data Center machines to continue accessing and storing a growing set of test management data. This may be a workable solution for manual shops, but if your team is moving towards higher percentages of automated testing, the amount of test data you’ll need to store will grow exponentially, requiring you to continue adding additional machines indefinitely.
Atlassian’s own documentation about Jira testing tools lists “known major problems with 3rd party apps in Jira.” Two test management apps included on the list are cited as the cause of stalls and Out of Memory Errors.
To better understand exactly where these performance issues are likely to occur, we ran an experiment with Jira Server and a common test management app. (The full results and details of the experiment are available here). Here’s a summary of what we found:
- Loading a few thousand test executions did not significantly affect the use of Jira. This indicates that Jira test management apps are a viable solution when only managing manual test cases at a small scale.
- At a small to medium scale, around 100,000 test executions, we found performance significantly impacted in the entire Jira application. In fact, Jira crashed multiple times when loading execution results from a standard to large sized test suite.
- When loading automation results into Jira, the Jira API was significantly impacted overall and latency was increased drastically on the entire application affecting non-testers, developers, business analysts and other users.
To learn more, read the full report.