Storage Constraints Quickly Cause Performance and Scalability Issues
At enterprise scale, Jira test management apps cause significant performance issues with Jira Server and Jira Data Center. Jira apps (formerly referred to as plugins or add-ons) lack built-in storage. This class of application relies on the Jira database to store the data it manages or creates. As data accumulates and the app consumes more space, the entire Jira instance becomes increasingly vulnerable to performance degradation, stalls and crashes, which impacts every Jira user throughout the organization – in as little as 14 weeks, according to our research.
Jira is a business-critical component of the software delivery pipeline, providing workflow management and a single source of truth across all stakeholders. A Jira crash brings software delivery workflows to a full stop. And reaching the scale that makes Jira vulnerable to crashes can happen much more quickly than you might think.
Atlassian’s documentation lists “known major problems with 3rd party apps in Jira.” Two test management apps included on the list are cited as the cause of stalls and Out of Memory Errors. According to Atlassian, an Out of Memory Error will halt email sends, prevent updates to the activity stream and cause the Jira application to crash. These issues might not be fully recognized during a POC or evaluation, but they quickly become apparent as more teams begin migrating test cases and artifacts to the Jira test management app from their legacy solution. See the diagram below to understand the difference between a Jira test management app and a dedicated, external test management solution, like Tricentis qTest.
Architecture differences between Jira test management apps & standalone test management:
The problem is accelerated as organizations begin scaling test automation to support agile development and DevOps delivery initiatives. Test automation significantly accelerates testing, so by nature it creates significantly more test result data, significantly more quickly, than manual testing alone. When a Jira test management app causes Jira to crash, testers can no longer store, access or reuse test cases, losing all visibility and traceability – which are often the main reasons they implemented test management in the first place. Moreover, all development work may grind to a halt. It is far better to begin a transformation with the right tools in place than to wait until mid-stream failures and delays cause the entire software development lifecycle – and revenue-generating releases that the business relies on – to stall.
Identifying the Tipping Point: When Do Performance Issues Arise?
To address the scalability issue, you could pay for more Jira Data Center machines to continue accessing and storing a growing set of test management data. This may be a workable solution for manual shops, but if your team is moving towards higher percentages of automated testing, the amount of test data you’ll need to store will grow exponentially, requiring you to continue adding additional machines indefinitely.
To better understand exactly where performance issues are likely to occur, we ran an experiment with Jira Server and a common test management app. Here’s what we found:
- Loading a few thousand test executions did not significantly affect the use of Jira. This indicates that Jira test management apps are a viable solution when only managing manual test cases at a small scale.
- At a small to medium scale, around 100,000 test executions, we found performance significantly impacted in the entire Jira application. In fact, Jira crashed multiple times when loading execution results from a standard to large sized test suite.
- When loading automation results into Jira, the Jira API was significantly impacted overall and latency was increased drastically on the entire application affecting non-testers, developers, business analysts and other users.
100,000 Test Executions
To give a clear indication of what 100,000 test executions looks like, consider the following advanced, but realistic scenario:
- 3 applications
- 8 builds a day (dev code commits – part of standard CI process)
- 30 smoke tests each run on the build (3,600 executions a week)
- 1,000 regression test cases run each week (3,000 executions a week)
- 5 performance test scenarios run each Build (600 executions a week)
- 1 weekly code quality scan result each week (3 executions a week)
- Daily static security scanning results (15 executions a week)
That’s a total of 7,218 test execution results that you’ll track just from automation per week.
You’re looking at about 14 weeks until you hit 100,000 stored test executions in this scenario. In other words, in this scenario, users could expect significant performance issues and crashes just three and a half months after adopting a test management app.
How Jira test management apps degrade Jira performance over time:
We know each project is different and you may or may not be automating as much or as frequently – but maybe you should be!
When scaling test automation, test result data is created on an order of magnitude greater than other systems because testing in the continuous integration process means results are added at a much higher rate than requirements, defects and other artifacts in Jira. For example, a sprint with 10 user stories could have hundreds of automated tests associated with each user story. Jira test management apps aren’t set up for this kind of scalability, so it’s easy to see how quickly this could get out of hand with multiple teams performing two-week sprints. The result is that Jira’s basic functionality such as reporting, search operations and type-ahead functionality are bogged down.
One Team’s Experience With a Popular Jira Test Management App
The R&D team at a large, national health plan began experiencing extreme performance issues across their entire Jira instance after just five months of using a Jira test management app. When the Jira instance became unusable, neither Atlassian nor the app company were able to provide a solution. Here’s how the head of testing describes the experience:
“We wanted to decouple Jira and test management because Jira users were being negatively affected by the app. After about five months of using the test management app, our Jira instance was deeply affected by it. We reached out, but [the app company] company refused to fix the underlying architecture problem because it would be too difficult fix. And then they stopped responding to our requests.
We chose to switch to Tricentis qTest – a dedicated, external test management tool that integrates with Jira in real time – because it doesn’t share the same database as Jira, so it gives us a more robust solution for the enterprise. In addition, Jira maintenance didn’t affect our test management tools and maintenance on the test management tool did not affect all Jira users.
When we were using [the test management app], we had to create a lot of custom fields in Jira and a lot of custom reporting solutions. In an effort to be lightweight, the Jira test management apps don’t typically come with robust reporting capabilities. But when you’re in the enterprise and you need to report data for multiple teams, you have to either create custom solutions or you use a tool like Tricentis qTest, which has been a big boon for us and our reporting processes.”
How Does Tricentis qTest Compare?
When Tricentis built integrations with Tricentis qTest and Jira, we wanted to ensure that the integration was built to scale with the needs of a team performing advanced test automation. So we did some of our own load and performance testing to ensure our integration would not have a negative impact on Jira users.
Because Tricentis qTest does not rely on the Jira database or data structures to house data, the only effects it has on the Jira system are:
- When an issue is updated, a webhook is used to notify qTest of the change
- When viewing an issue, Jira is also loaded with a view of relevant qTest testing data
We tested across various setups of Jira including Jira Data Center, Jira Server and Jira Cloud. Here is a sample of our results:
- Generally, we found no significant slowdown to display or
save Jira issues
- It took about 200-300ms to load the additional test result data from the ‘split-view page’
- It took 600ms on average to show the test result data on the full screen ‘detailed issue page’
- No slowdowns were introduced in Jira reporting, dashboards, searching, APIs, etc.
We did three rounds of sampling of our tests using Tricentis Flood, and then averaged the results. Below is an example of one round of performance testing and a summary of response times gathered during three rounds of testing.
A summary of response times with and without the qTest integration activated:
What Other Enterprise Features do Jira Test Management Apps Lack?
Beyond the ability to support agile testing at scale, there are several requirements associated with testing within a DevOps pipeline that Jira test management apps just can’t deliver – from granular user and project management, to advanced reporting and analytics to flexible integration options that embed quality throughout the software delivery pipeline. Organization-wide transformation demands the performance, reliability and scalability of an enterprise-grade test management tool.
Consider the following limitations:
Limited Integration Options
These test management apps typically offer only one method for integration. These are often paid add-ons (i.e. Zephyr Zapi). This lack of flexibility makes it much more difficult to integrate the testing toolstack into the rest of the software delivery pipeline, significantly limiting the effectiveness of testing in a highly integrated DevOps environment.
Tricentis qTest offers out-of-the-box integrations with leading tools such as Jira, Confluence, Tricentis Tosca, JUnit, Tricentis Flood, Postman, CA Agile Central, VersionOne and many more, and testers are offered multiple integration options, including APIs and SDKs, a dedicated test automation management tools and CI pipeline integrations with Jenkins and Bamboo.
Limited Reporting & Analytics
If you’re trying to transform testing, reporting on your progress is a must. Typically, customers who select Tricentis qTest are doing a whole lot more than just purchasing a tool. They’re changing their approach to quality to support agile development, DevOps delivery and digital transformation initiatives. Advanced reporting capabilities are critical to demonstrating the return on your investment and tracking improvements in quality for both individual projects and across the organization. Jira test management apps typically lack executive-level reporting and cross-product reporting capabilities, making it difficult for R&D teams and business stakeholders to get the visibility they need.
Tricentis Reporting provides easy-to-use interface that lets you track quality at the project, team or organization level, across testing tools and frameworks. With Tricentis Reporting, testers can report on Jira artifacts, such as requirements & defects, for projects that are linked in Tricentis qTest.
Limited Enterprise Features
Tricentis qTest offers the enterprise-grade administration & management features that these test management apps lack – and that are critical for an advanced testing practice in an enterprise environment. These include:
- Advanced user administration including user, group & project management, SSO integration and role-based permissions
- Ability to create project templates
- Parameterized testing data
- Embedded test cases
- Version control
- Test case sharing
- E-signature support
- Event-driven approach (like all modern, enterprise-grade tooling)
- A dedicated database (that is not Jira’s) in addition to Tricentis’ qTest horizontally scalable cloud or on premise offerings means Tricentis qTest can easily support scaling to 10,000+ users and millions of test cases and executions
Test management apps are a good solution for teams performing lightweight, primarily manual testing – Atlassian recommends them for this use case. But large organizations with multiple projects across agile teams will require a dedicated test management application with the ability to provide dedicated support.
Tricentis’ primary focus is modern software testing solutions that support digital transformation at scale. We are a partner in our customers’ transformation initiatives – not a vendor dropping off a tool on your doorstep. Tricentis’ dedicated support teams offer assistance with implementation, ongoing support, a robust knowledgebase with video training, access to a community forum and more. To learn more about Tricentis, visit the website or contact us to request a demo.