Xray alternative

An Xray alternative for Jira teams

Keep Jira for bugs and stories. Move your tests to a place built around one question: is the release ready?

By the TestPlum teamChecked

Short answer

TestPlum is an Xray alternative for Jira teams that want their tests outside the issue queue and a Ready or Not ready verdict for each release. It imports Xray tests as JSON or CSV, including Jira's CSV export, and each test keeps its Jira key. Bugs still go to Jira and stay in sync both ways, and you pay per tester rather than for every user on the Jira site.

Who should switch, and who should stay

Xray describes itself as test management built inside Jira. That is a real strength if your team wants tests to be Jira issues. TestPlum takes a different approach: tests live in TestPlum, and Jira keeps the bugs and stories.

  • Switch if most people on your Jira site never touch tests, and you would rather pay only for the people who do.
  • Switch if you want automated and manual results to add up to one release answer, with flaky passes and retries weighed honestly.
  • Stay if your team wants tests, test plans and test sets to be Jira issues, searchable with JQL next to everything else.
  • Stay if you work mainly in Cucumber. TestPlum keeps a Cucumber scenario as text in the test's description; it does not turn Gherkin into steps.
  • Stay if you need step-by-step results history. Each test's result in an execution comes across, but the status of each step stays in Xray.

TestPlum and Xray side by side

For Xray we list only what Xray, its Atlassian Marketplace listing and Atlassian's own documentation say. Where they don't say, check their docs.

TestPlum compared with Xray
CapabilityTestPlumXray
Where tests liveIn TestPlum, linked to JiraInside Jira
Pricing modelPer tester; viewers and Jira users not countedMarketplace app; on Jira Cloud the license tier matches the Jira site's user tier
Free optionFree plan: up to 3 testers, 1 project, no time limitFree trial on the Marketplace
EditionsFree, Team, Business, EnterpriseStandard and Advanced
Import from XrayJSON, GraphQL results, Test Case Importer CSV or Jira CSV; Jira keys keptNot applicable
CI resultsJUnit XML, CTRF and Allure by one curl stepConnects to JUnit and CI tools such as Jenkins, GitHub, GitLab and Azure DevOps
BDDCucumber scenarios kept as textCucumber tests in Jira
Release verdictReady or Not ready from agreed checks, with plain reasonsSee their docs
Pass weighed by evidenceTrusted pass rate: retries, flaky passes and passes with no assertions count partlySee their docs
Failure grouping and sortingGrouped by cause; sorted as bug, broken test, environment or flakySee their docs
Flaky-test quarantineYes, with reason, owner and expirySee their docs
Jira bugs and storiesBugs sent to Jira with two-way status sync; stories imported by JQL as requirementsNative to Jira

What the Xray importer brings over

Upload an export under Tests, Import, Xray. TestPlum recognises Xray's bulk JSON, the JSON its GraphQL API returns, a CSV laid out for the Test Case Importer, or Jira's CSV export of the test issues. You see what will happen before anything is saved. The Xray migration guide has the full field map.

  • Test repository folders become folders, in the same nesting.
  • Steps keep their action, test data and expected result together.
  • Labels become tags, Jira priorities map to TestPlum priorities, and requirement links become references.
  • Each test keeps its Jira key, such as SHOP-12, so a newer export updates the same tests.
  • Attachments and images on the test issue can be copied with a Jira API token you give once and that is never stored.
  • Executions come across as past results: each becomes a signed-off run dated as it happened. Test plans and test sets stay in Xray.

Send CI results with one curl step

If your pipeline already produces a JUnit report for Xray, the same file works for TestPlum.

TestPlum takes JUnit XML, CTRF JSON and Allure results at one address and works out the format from the content. CTRF covers reporters for Cypress, Jest, Vitest, pytest and more. Retries in CTRF and Allure are kept as separate attempts, so a test that passed only on its second try shows up as flaky instead of green.

Open Runs, Connect CI, pick your test tool (Playwright, Jest, Vitest, pytest or any JUnit XML) and your CI (GitHub Actions, GitLab CI or another), and create a CI key. The key is shown once and can only send results and read the answer for that project. The page waits for your first run and shows it as soon as it lands. Nothing to install: the step is plain curl.

One CI step: send a JUnit XML report (a CTRF or Allure report goes to the same address)
curl -f -sS -X POST \
  -H "Authorization: Bearer $TESTPLUM_CI_KEY" \
  -H "Content-Type: application/xml" \
  --data-binary @junit.xml -o /dev/null \
  "https://app.testplum.com/api/v1/projects/<project-id>/automation/report?release=%40next&run_name=CI%20%23<build>"

A release verdict your whole Jira site can see

Viewers are free, so developers and product managers can open the verdict without a paid seat. Read how it works on the release verdict page.

  • Ready or Not ready, from checks you agree in advance: runs in scope, completion, pass rate, trusted pass rate, critical tests failing or blocked, open critical bugs and requirements verified.
  • Each failing check reads as one plain sentence, such as “Only 50% of tests pass. Needs 95%.”, with one button to the page that fixes it.
  • A pass counts by what it proves. A pass on retry, from a flaky test or with no assertions counts only partly, so a green suite of weak tests cannot make a release look ready.
  • Your build can print the answer in its log, or stop when the release is Not ready.

Failures sorted by cause, bugs filed in Jira

When automated tests fail, TestPlum sorts out why before anyone opens a ticket. More on failure triage, and on the Jira connection.

  • Failures that share a cause are grouped, so ten tests failing for one reason are one problem.
  • Each group is sorted as a product bug, a broken test, an environment problem or a flaky test, by visible rules, with the reasons and a confidence.
  • File one bug per cause and send it to Jira or GitHub. Its status stays in sync both ways.
  • Flaky tests can be quarantined with a reason, an owner and an expiry. They stay out of the verdict counts until a fix is confirmed, except that a quarantined critical test keeps the verdict at Not ready.

How the pricing models differ

Xray is sold through the Atlassian Marketplace in Standard and Advanced editions, with monthly or annual billing and a free trial. Atlassian's documentation says a cloud app's license tier matches the number of users licensed for Jira on the site. In practice that means the app is priced for everyone on the Jira site, not only the people who test.

TestPlum is priced per tester. Free covers up to 3 testers, unlimited viewers and one project with no time limit. Team is $19 and Business $35 per tester per month, billed yearly. Viewers, such as developers, product managers and managers, are free on every plan, and people in connected tools such as Jira or GitHub are never counted. See pricing.

For example, a Jira site of 200 people with 10 testers pays TestPlum for 10 testers: $190 a month on Team, billed yearly. For Xray's price at your Jira tier, use the pricing tab on its Marketplace listing. Paid TestPlum plans are set up with the TestPlum team for now.

How to move from Xray

  1. Create a free workspace. Sign up and create a project. No Jira or Xray credentials are needed to import.
  2. Export your tests. Use Xray's JSON, its GraphQL API, the Test Case Importer CSV or Jira's CSV export of the test issues.
  3. Check and import. Upload it under Tests, Import, Xray. Fix anything the check lists, then import.
  4. Bring executions. Import Xray's JSON results as past results, matched by Jira key.
  5. Connect Jira. Connect your Jira project so bugs go there and stay in sync, and import stories by JQL as requirements.
  6. Connect CI. Add the curl step from Runs, Connect CI, using the JUnit report you already have.

Frequently asked questions

Can I import Xray tests?

Yes. Upload Xray's bulk JSON, its GraphQL results, a Test Case Importer CSV or Jira's CSV export under Tests, Import, Xray. Folders, steps with test data and labels come across, and each test keeps its Jira key.

Does TestPlum work with Jira?

Yes, but it is not a Jira app. Bugs are sent to a connected Jira project and their status stays in sync both ways, and Jira stories can be imported by JQL as requirements.

Do I pay for every Jira user?

No. TestPlum is priced per tester only. Viewers and people in connected tools such as Jira are never counted.

Is there a free Xray alternative?

TestPlum has a free plan for up to 3 testers with unlimited viewers and one project, with no time limit. The Xray import is included.

Are Xray executions imported?

Yes. Import Xray's JSON results after the tests. Each execution becomes a signed-off run dated as it happened, matched by Jira key. Test plans, test sets and step-level statuses stay in Xray.

What happens to Cucumber tests?

The scenario is kept as text in the test's description, and the import check lists which tests this applies to. TestPlum does not turn Gherkin into steps.

Sources

Xray and Jira are trademarks of their owners. TestPlum is not affiliated with or endorsed by them; the names are used only to compare and to describe what TestPlum can import.

Find out whether your next release is ready

Create a workspace, import your tests or write a few, and record a run. Everything on this site is in the product today.