We’ve recently fielded a lot of questions about using Jira for test case management. From how to use Jira for testing to which solutions best complement a Jira process, both Jira and testing seem to be the topics of the hour these days.

This popularity is not necessarily surprising given a recent Tricentis poll in which 66% of respondents reported having used Jira for the past several months or years.

So what exactly do you need to know when it comes to using Jira for test case management? This guide will walk you through everything from soup to nuts, including:

1. Challenges of using Jira for testing
2. How to customize Jira for testing
3. Pros and cons of using strictly Jira for test case management
4. Options for integrating testing tools with Jira
5. Tips for finding the best solution mix for your needs

 

A quick background on Jira

For those who don’t know, Jira is an issue and project tracking software from Atlassian. Currently, it’s the number one software development tool used by Agile teams, and as Agile adoption continues to grow, more and more organizations are introducing Jira. The recent IPO as well as the massive wave of companies leaving legacy ALM providers have only fueled the widespread adoption of Jira.

Although JIRA is designed for issue and project tracking, many teams are using it for test case management so that development and testing can stay in one system. But how exactly is JIRA set up for testing? Let’s take a look.

Challenges of using Jira for testing

Going back to the question of how JIRA is set up for testing, the short answer is that JIRA isn’t. The reality is a lot more complicated, especially given how many organizations are using it for testing today.

When you begin looking at how you can build a testing process in Jira, the most common place to start is with an issue type. You’ll see that Jira has several different types of issues (Bug, Epic, Improvement), but nothing specifically for tests. This is because Jira is not a test case management solution. It’s really meant for issue tracking.

You can customize Jira in certain ways to use it for some test case management processes, but these are makeshift customizations and they have limitations as a result. For example, every workflow built inside Jira has an endpoint of “done.” But in testing, even if you complete a test, that doesn’t necessarily mean that you’re actually done with that test.

This setup means that if you customize Jira or certain issues for test case management, you’ll always find yourself moving to a done status when you might not be “done” once a test executes. What you really want is to be able to use that test case for multiple types of testing, whether it be for regression testing or running the same test case under various parameters and configurations to validate a new feature. However, since Jira issues are always meant to be pushed to “done,” you are limited in the flexibility to reuse the issue for testing.

Atlassian’s own documentation on Jira says that it can work for manual testing, acceptance testing and so on, but it clearly warns that there is no testing-specific functionality built naively into Jira. Also, if you are using any type of automation, you need to use another solution.

Even with manual and acceptance testing, there are some limitations. Let’s say that Jira does a really good job of running a manual test case. It might work one time or for a brand new feature, but then you move it to the done status and what happens next? What if you want to do regression or session-based testing? How are you going to reuse those objects across projects, assign multiple ticket items and show that coverage? Due to the way Jira is set up, you can’t.

How to customize Jira for testing

Despite the limitations posed, many teams are using JIRA for test case management, and they’re typically doing so by customizing issues in one of two ways: adding a “test case” issue type, or using the “user story.” Let’s take a closer look at how each of these options works:

Deep dive: Adding a “Test Case” issue type

To customize Jira for test case management by adding a “Test Case” issue type, you need to take the following steps:

1. Create a “Test Case” issue type
2. Add in the steps required to complete an expected outcome
3. Make that test case the parent issue for the testing you need to do
4. Create a subtask and label that subtask as “Test Run” in order to execute the test
5. Put in the outcomes, the affected versions, the results of that and the assignee who’s running the test within the “Test Run” subtask

While this approach works in theory, in reality it presents several challenges:

• Rerunning tests: If you need to rerun a test execution or test a new version, you need to add more test runs as subtasks, and you have to add a subtask every single time you want to log the history of that test case.
• Reusing tests: With test case management, you often need to reuse test runs and have visibility into logs of those test runs, but you can’t reuse subtasks in JIRA because they are marked as “done”.
• Creating coverage reports: Because all the subtasks are assigned under one parent issue, you can’t group several test runs to different configurations and show a coverage report.

Deep dive: Using the “User Story”

Alternatively, you can customize Jira for test case management by tweaking the “User Story”. Here’s how this approach works:

• Create a User Story (this will act as the test case in the same way that the new parent issue did in the first option)
• Add a subtask that acts as a test case or a test run that refers back to that User Story
• If all the subtasks pass, then the User Story is ready to go into production

Once again, this approach poses challenges because the subtask acts as the test case and the test run. Specifically, it poses challenges around:

• Reusing tests: If you ever want to use the subtask again, for example in a regression cycle or as a link to another User Story, it becomes complicated since the User Story has likely been marked as “done”.
  • Note: With Jira’s workflow, all issues that are marked as done get closed out. If you have to re-open a subtask and then convert it to a test case template or a test case type, you end up with a lot of overhead maintenance.
• Aligning test cases: Due to this setup, if you have multiple test cases that link to multiple user stories, aligning test cases becomes extremely difficult.

Pros and cons of using strictly Jira for test case management

As the above examples illustrate, while Jira is not designed for testing you can customize it to fulfill certain test case management needs. But, should you do this? Let’s consider the pros and cons of using strictly Jira for test case management:

Pros of using Jira for test case management:

• Ability to create custom issue types such as test cases
• One workflow for QA, developer and testing actions
• Ability to use existing reporting (and Jira does offer excellent reporting)
• Ability to use an already purchased and known tool (if you already have Jira in place)
• Good for one-time manual execution

Cons of using Jira for test case management:

• No testing specific functionality (this is something that Atlassian itself calls out)
• Only integrates with testing frameworks via the CI server
• Inability to create a test case coverage report that covers manual execution, automation execution and session-based execution
• No traceability reporting between issues and test case coverage
• No way to run executions more than once, which means significant duplication of effort
• Limitations around test cycle and suite execution, test step execution status, version control and environment configuration

All of this means that things you would normally do with a test case and environment execution tool, such as grouping testing cycles and test executions, running test configurations and handling version control, are not supported. This makes you think: Is Jira really built for testing or are you forcing the issue (no pun intended, since everything in JIRA is an issue)? In most cases, you’re forcing the issue and there is likely a better way.

Options for integrating testing tools with Jira

Because Jira is not designed for testing, it poses many limitations when it’s used for testing purposes. Arguably the biggest limitation is the inability to reuse and centralize testing efforts.

So if you’ve made the transition to Jira and your developers love it, what can you do? Is there any way for your test team to get value out of Jira? Integrations can help.

Because Jira is not designed for testing, it poses many limitations when it’s used for testing purposes. Arguably the biggest limitation is the inability to reuse and centralize testing efforts.

There are two types of integrations you can use to add testing functionality to Jira:

1. Add-ons, which are internal integrations that live inside the Jira application and extend some of the test case functionality
2. An external integration with a dedicated test case management tool

 

Let’s take a closer look at what each of these integration options entails:

What about Jira apps or add-ons?

Add-ons are available via the Atlassian Marketplace. A few commonly used add-ons for testing include:

• Xray for Jira and Zephyr for Jira for manual test execution (Xray also offers a bit of functionality around BDD testing)
• Behave Pro and Tricentis qTest Scenario for BDD, TDD and ATDD testing (these are best for smaller teams or those that are embracing more of an Agile 2.0 space inside of JIRA)

These types of add-ons extend the functionality of Jira to make it better equipped for test case management. For example, Xray automatically adds a “Test” issue type to the list of issues and allows you to add test case steps to the Test issue type. This setup alleviates the problem of creating a custom issue type that requires you to add in a bulk section that includes completed steps and expected outcomes (which in turn leaves you with huge chunks of testing that you have to do all at once and the inability to put in line by line where the steps failed, where it was executed and what your actual execution was).

Meanwhile, if you’re trying to do BDD testing, it’s hard to follow the Gherkin “given, when, then” syntax in JIRA alone, but an add-on like Tricentis qTest Scenario makes it possible. Tricentis qTest Scenario allows you to import your dot feature files along with your scenarios as issue types so that you no longer have to create standard issue types. Instead, you can add in the Gherkin style test scripting language in the scenario piece, where you can then create a dot feature file. You can also create scenario files that attach to those dot feature files inside of JIRA, which allows you to add them to a User Story.

Pros of using add-ons

In general, the testing add-ons try to bring together the groups of testing occurring in Jira so that you can pull all of the test cases you have into cycles or suites. They also allow you to see the latest execution results of the test cases for each grouping, which creates visibility into the grouping of test executions.

Other pros of using add-ons to improve the testing functionality of Jira include:

• Maintaining a similar look and feel to Jira
• Getting to use the dashboards inside of Jira
• Simplifying purchasing by handling it online through the Atlassian Marketplace
• Receiving access to testing specific functionality
• Gaining the ability to link test objects to other issues

Cons of using add-ons

While the add-ons do help a lot, you’ll still be a bit strapped in terms of what you can and can’t do with your text execution because the add-on functionality is limited based on what the Jira infrastructure allows.

For example, the add-ons create a bit more of a manual process than you would have with a dedicated test case management solution. Additionally, everything you do inside the add-on is limited to the particular project at hand, meaning you can’t put the same issue in multiple projects.

The add-ons also present some version limitation when you’re creating a test case. Typically, if you create an issue and go to run the issue, the run is simply a duplication of the test case. Let’s say you have five steps in a test case that’s in a cycle to be run and you add another step. Jira will automatically push that data over into every test case run, meaning you can’t keep different versions. With Tricentis qTest and some other tools, you can actually keep track of different versions of test cases and run past versions when you execute, but you need to have a one-to-one version history of that test case execution in order to do that.

Finally, when you’re using an add-on, a test case equals a Jira issue. It may not appear that way, but that’s how it is with all the add-ons, and that setup poses limitations.

Other cons of using add-ons to improve the testing functionality of Jira include:

• Zero or limited availability to reuse test cycles (you might be able to clone a test cycle or you may have to move test cases into an ad hoc test cycle that’s not attached to a release)
• No logical folder structures
• Unknown change history for test case versions and limited visibility into test execution history
• Inability to share test case objects across Jira projects, which makes scaling difficult and creates a lot of duplicate work since you can’t run the same test with different variables
• Limited scalability for storing a high volume of test result data, which often becomes a problem when teams transition to automated testing

Deep dive: Dedicated test case management solution

Most testing teams struggle to be very efficient. Often it’s because they’re simply not using the right tool. A dedicated test case management tool can help improve efficiency, and there are many that now offer some level of external integration with JIRA in order to bring more testing-specific functionality to your testing process while still allowing you to use JIRA.

Most testing teams struggle to be efficient. Often it’s because they’re simply not using the right tool.

What do test management tools do?

Test management tools:

• Consolidate all of your automated and manual execution: A lot of the add-ons focus on just manual execution, BDD or TDD, but if you’re doing a Selenium automation or a custom framework, you’ll want to consolidate all of your executions in a way that also communicates with JIRA. A dedicated test case management tool can help you accomplish that objective.
• Offer enhanced execution setup and the ability to reuse executions: Add-ons can group test cases into cycles, but they can’t put testing cycles inside other cycles or have a cycle that tests suite execution. Dedicated test case management tools, on the other hand, can do that. These tools also allow for scalability by making it so that you don’t have to go back to the source record and copy and change test cases.
• Display the entire history of execution inside Jira: By displaying the entire history of execution inside Jira, a dedicated test case management tool can help you determine how a test case performs through testing phases. This view helps you easily see things like why a test case failed.
• Allow you to share test cases with multiple projects: Dedicated test case management tools can also eliminate the isolation of test cases by making it easy to share test cases with multiple projects. For example, with Tricentis qTest, you can have multiple test runs of the same test case and each test run can have a different configuration, environment setup, results, assignees, etc., but they can all be tied to the same test case. This setup allows you to use the test case as a template instead of having to duplicate it over and over again.
• Offer enterprise scalability: Unlike Jira apps and add-ons, Tricentis qTest does not rely on the Jira database or data structures to house data. To learn more about the storage limitations of Jira, read the results of our experiment with Jira Server and a common test management app.

Pros of using a dedicated test case management tool

By offering the functionality described above, integrating a dedicated test case management tool can go a long way toward improving testing within Jira. Pros of this approach include:

• Access to enterprise test features like configuration manager, CI integrations with Jenkins and Bamboo and automated testing support
• Ability to use test case repositories that are customized based on templates
• Ability to reuse test cases and share objects across different projects
• Support for multiple testing strategies

Cons of using a dedicated test case management tool

Despite all the value integrating a dedicated test case management tool with JIRA can provide, it still has its limitations. Cons of this approach include:

• Handling constant database synchronization between the tool and Jira (more on this to follow)
• Managing permissions for artifact mapping
• Working around browser restrictions (e.g. HP requires Internet Explorer)
• Missing a full integration with Jira
• Working around varying degrees of open source friendliness among dedicated test case management tools

With these downsides in mind, there are a couple points to consider to help determine whether or not the pros of an external integration with a dedicated test case management tool outweigh the cons:

• Does your team have a wide range of testing methodologies? If you’re a small team that’s only doing manual testing and there’s no push for automation, an add-on or Jira alone might work. However, if you have a wide range of testing methodologies, you likely need a dedicated test case management tool.
• Does your test strategy warrant an enterprise testing tool? If it’s something you need, then an external integration is the way to go. If it’s simply a nice to have, an add-on or Jira alone might work.

Getting the best of both worlds

While a dedicated test case management tool offers the best experience for testing inside of Jira, it typically still poses some downsides. However, it is possible to get the best of both worlds between an enterprise test case management tool and Jira.

The key to achieving this balance is understanding that not all integrations are created equal and identifying the type of solution that offers the optimal integration experience for your needs. For example, most test solutions use a scheduled sync, but Tricentis qTest Manager has a real-time integration that uses webhooks.

With a scheduled sync, you have to map your entire test case management tool schema to your JIRA schema, which means you constantly have to pull data and use resources to keep the system current. A scheduled sync is also prone to delays, poor performance due to APIs constantly trying to figure out what’s been updated and data corruption if any schedule collisions occur. All of this requires a lot of resources to manage the integration.Meanwhile, with a real-time sync, you get a straight mapping between a project (and its issues, whether they’re User Stories, Epics, subtasks, etc.) and JIRA, and you can then show test case coverage against those issues. This real-time sync offers a direct submission into JIRA for defects, browser automation and embedded reporting, plus it ensures data accuracy since JIRA is always the source record for user stories and defects.The real-time sync option keeps both developers and testers happy because it allows everyone to keep doing all of their updates inside of Jira and only requires you to pull the issues that you care about into the test case management tool, where you can then create test cases and test runs and map those to the requirements from Jira. Once you’ve executed those runs, you can simply push the defect back out to Jira to interact with it and start working on a resolution. This allows everyone to keep doing all of their updates inside of JIRA and only requires you to pull the issues that you care about.With this type of integration in place, your developers can stay inside of Jira (rather than going to another tool to see everything) and your testers gain access to the enterprise features that are missing from a lot of the add-ons that are available — all without a complicated, time-consuming integration to manage. In other words, you get the best of both worlds.

Which option is right for your organization?

Ultimately, which mix of solutions — Jira only, Jira plus an add-on or Jira plus an external integration — is right for your team depends a lot on the size of your team and the types of testing your doing.

To help you determine the solution mix that best fits your needs, we’ve mapped out our recommendations in the following chart: