There are two big names in the Atlassian Marketplace for enhancing Jira test management: Xblend’s Xray, and SmartBear’s Zephyr.
Both provide a new layer within Jira that enables consistent test management and tracking, and both are easily integrated. But which is the right plugin for your Jira test management needs?
Let’s find out.
What is Xray?
Xray Test Management for Jira is an Atlassian Marketplace app, designed to help Jira users to plan, test, track and release software developments.
Created by Xblend, Xray was initially developed c. 2013. In the following decade, Xray has seen several major revisions, including an Atlassian Cloud offering released in 2018.
Xray offers a rich connection with Atlassian Jira, as well as integrations with additional CI/CD tools like Jenkins, GitLab, and Atlassian’s Bamboo (though, we should add, this comes with some heavy caveats, which we will touch on later in this article).
The app includes several new work item types (previously known as ‘issue types’), including the obvious Tests, Test Sets and Executions, alongside Preconditions, Plans, and Sub Test Executions.
As all objects within Xray are Jira work items, it’s easy to configure reporting using default Jira features like dashboard. This is worth noting, as the reporting tools provided in-app are a little lacklustre.
Xray key features
Alongside the usual capabilities you would expect from a test management solution, Xray also includes the following handy features:
Easy grouping of tests using folders and/or test sets.
A built-in test execution timer that can start automatically when test execution begins.
Deeper per-project customisability. Xray work item types can be removed from individual projects, mapped as ‘defects’, and test step formats can be expanded to more than the standardised “Requirement, Action, Outcome”.
Test run archival functionality, reducing system ‘bloat’ and speeding up view and report generation. Configurable between 1 week – 2 years.
Xray does however, have with a few key drawbacks worth highlighting.
We briefly mentioned this earlier: Xray’s Atlassian Bamboo connector, listed on the marketplace, is no longer supported (9.X+), and does not have a Data Center equivalent.
Other connectors for tools such as Jenkins and GitHub, require Xray enterprise licensing, which may be more costly for your organisation.
As mentioned, Xray’s reporting capabilities are a little light. To generate or export some reports, you’d need Xporter (an additional plugin which costs extra) to enable this.
Xray pricing
Xray offers two subscription tiers depending on your organisation’s requirements: Standard and Enterprise.
Enterprise is, as you’d expect, more expensive – but it does offer significant benefits to large businesses, enabling testing features such as versioning, remote job triggering and cross project reporting.
Xray Enterprise can also provide an organisation with more confidence and scalability via enhanced 24/7 SLAs and unlimited storage. Standard, in comparison, is restricted to 100GB of attachments within Xray objects.
Looking for support with Jira test management best practice, integrations, or your techstack as a whole? Talk to us today to review your techstack strategy and get a free 30-minute session to explore optimisation and practical recommendations you can take away!
What is Zephyr?
SmartBear’s Zephyr, formerly Test Management for Jira (TM4J), is also an Atlassian Marketplace app.
Originally founded in 2007, SmartBear released a Zephyr Cloud app version in 2016. There are two core products under the Zephyr umbrella: Zephyr Essential (previously known as Zephyr Squad), which is effectively a ‘light’ version of the tool, and Zephyr (formerly Zephyr Scale), which is available on a Standard or Advanced tier.
Within Jira, SmartBear’s software is implemented very similarly to Xray, using a combination of Jira features (work item types for tests and added custom fields), and bespoke-developed menus and views.
We would caution that its UI can be cumbersome to navigate if you don’t know quite what you’re looking for and where. On the other hand, however, its built-in reporting views are more easily digestible, and by opting for Zephyr over Zephyr Essential, the reporting capabilities expand immensely.
🌎 A note on compliance and data residency
Zephyr and Xray both store relevant test data outside of the Atlassian Cloud.
However, with Zephyr, there is limited configuration for how and where this is located. The tool does not provide data residency locations – all data is stored in AWS in the US.
If data residency is a specific requirement for your organisation, perhaps due to GDPR constraints, Xray offers data residency in the US, EU (Germany), and Australia.
Zephyr key features
As with Xray, Zephyr provides the capabilities you’d expect from a test management app. These include:
Grouping of tests
Report generation and export to excel
Archiving function for test executions/runs
Configuration-wise, Zephyr Essential is not as extensive as Xray, providing only ‘Tests’ as a work item type. Test executions share a 1-1 relationship, and are not represented as their own work items or sub-tasks under a test. We’d caution that this can hamper options for creating more in-depth reporting, and could force an increased dependence on the tool, as the data cannot be easily converted if you wished to uninstall the app, or migrate to an alternative solution.
Zephyr also has the capability to integrate with external tools such as Jenkins and Bamboo, but the process is manual, and we understand that there is currently no installable add-on that will allow seamless integration between both tools.
Therefore, the integration must be configured on a per-build basis, using pre-provided REST API scripts that require adjusting to your specific infrastructure setup. Whilst this may add better granularity on what builds are integrated where, and how test executions are submitted, it is more time-consuming.
Need an expert eye to review your techstack? From auditing your configuration and plugins, to recommending practical steps to optimise your security, performance and output, talk to our experts here at AC.
Jira test management apps: Which is right for you?
Let’s wrap our discussion up. We’ve provided an overview of both tools, and outlined their various pros and cons. So, what’s the verdict?
If you’re a small to medium-sized business, perhaps introducing a testing strategy for the first time or using an existing platform you’re unhappy with and looking to consolidate into the Atlassian environment, Zephyr Essential may be the way to go. It is generally cheaper for organisations of less than 500 users, and its feature set is sufficiently similar enough to Xray to provide solid value for money.
Once you’ve outgrown Essential, either through user growth or work complexity, you can look to adopt Zephyr (Standard or Advanced level) and harness the expanded capacities it provides.
If you are an enterprise-level organisation, meanwhile, requiring unlimited storage, strong customer support, and tight pre-developed integrations with your existing CI/CD stack, Xray may be the better choice for you.
Xray’s standard offering has the greatest number of installs, and the highest review rating compared to either Zephyr Essential or Zephyr. From our perspective, the UI is more intuitive and user friendly, and as a tester, is easier to utilise. We also rate the way it uses Jira work items to represent all testing objects. For smaller organisations, this may add an unnecessary overhead (which is, again, why we’d recommend Zephyr Essential for SMBs) but enterprises will benefit from the richer level of customisability this provides.
Enhance your Jira testing practice with expert guidance
Are you or your organisation having difficulties implementing a consistent testing practice within Atlassian? Or had you already adopted one of these tools and we’ve swung you towards considering the other? You’re in luck!
AC are not only Atlassian Platinum Solution partners with extensive knowledge of the Atlassian Cloud Platform, but we also have first-hand experience of using and implementing both Xray and Zephyr.
With years of expertise providing consultation on general testing best practises, as well as assisting with software adoption and migration, turn to us with your Jira test management queries.
