CTRF integration
A CTRF report viewer and dashboard that keeps every retry
CTRF gives every test tool the same JSON report. Send it to TestPlum and each retry becomes its own attempt, so a test that passed on the second try is shown for what it is.
CTRF, the Common Test Report Format, is an open standard for JSON test reports. Upload a CTRF file to TestPlum with curl and a project CI key; retry attempts become separate results, failures are grouped by cause, and the project answers Ready or Not ready.
What is CTRF?
CTRF stands for Common Test Report Format. Its site calls it an open standard for JSON test reports: one structure for results, whichever test framework produced them. Its site lists reporters and tools for Playwright, Jest, Cypress, Vitest, pytest, Mocha, WebdriverIO and many more.
Each test in a CTRF report has a name and a status. It can also carry the file path, the suite, a message and trace, steps and earlier retry attempts. TestPlum reads all of those.
Upload a CTRF report
Add a CTRF reporter to your test tool, following its own instructions, and note the file it writes. In TestPlum, open Runs → Connect CI and press Create CI key. Save the key as a secret named TESTPLUM_CI_KEY. Then run this after your tests.
The Connect CI page builds its step from JUnit XML. This command is the same step pointed at a CTRF file: it sends JSON and names the format.
# After your tests, with TESTPLUM_CI_KEY set to the key.
# Change ctrf-report.json to the path your CTRF reporter writes.
curl -f -sS -X POST \
-H "Authorization: Bearer $TESTPLUM_CI_KEY" \
-H "Content-Type: application/json" \
--data-binary @ctrf-report.json -o /dev/null \
"https://app.testplum.com/api/v1/projects/<project-id>/automation/report?format=ctrf&release=%40next&run_name=CI%20%23$BUILD_NUMBER"
curl -f -sS -H "Authorization: Bearer $TESTPLUM_CI_KEY" "https://app.testplum.com/api/v1/projects/<project-id>/status?format=text" | tee testplum-answer.txtHow CTRF fields map to TestPlum
A test TestPlum has not seen is created in an Automated suite, with sections from its file path and suite. The next report with the same automation ID adds results to the same test.
| In the CTRF report | In TestPlum |
|---|---|
| filePath, suite and name | The automation ID: filePath > suite > name, with the parts that are present |
| Retry attempts | Earlier attempts, recorded before the final result |
| passed, failed | Passed, failed |
| skipped, pending | Skipped |
| other | Blocked |
| message and trace | The error, kept on the result |
| steps | Step outcomes on the result |
Why retries matter
A test that fails and then passes on retry is green in most summaries. In TestPlum it is recorded as failed, then passed. That pass counts for less in the trusted pass rate, and the flip feeds the flaky score, so you see the problem before it hides a real bug.
What the dashboard gives you
- Run history. Each upload becomes a run named after the build, such as CI #412, next to your manual runs.
- Flaky detection and quarantine. Tests whose results flip between pass and fail are suggested for quarantine. A quarantined test is left out of the verdict counts until it passes enough times in a row, or the quarantine expires.
- Failure grouping by cause. Failures with the same normalized error become one group, sorted as product bug, broken test, environment or flaky, with the reasons shown.
- Trust-weighted pass rate. Each pass counts by how much it proves. A pass on retry, a flaky test or a test edited after the run counts for less, so a 100% pass rate can still fail the trusted pass rate check.
- A release verdict in plain words. The project answers Ready or Not ready, with each reason in one line and a link to where to fix it.
- Answer history. TestPlum remembers when the answer turned and what changed: up to five changes, newest first, with the reasons that were new or fixed.
Security: a CI key locked to one project
The key you paste into CI is a CI key, made under Runs → Connect CI. It belongs to one project and can do three things there: send results, run the release gate and read the status. Any other request with it is refused with 403, and a revoked key gets 401.
A CI key acts as the person who created it, so it can never do more than they can. Only a signed-in person who can run and edit tests can create one. It lasts a year, a project can have at most 20 live keys, and creating or revoking a key is written to the audit log.
Store it as a masked secret in your CI, never in a file you commit. If it leaks, revoke it on the same Connect CI page and make a new one.
Troubleshooting
- 400: not a CTRF report. The file is not valid CTRF JSON. Check the path in --data-binary and open the file to see that it has a results object with tests.
- 422: the report has no tests. The tests array is empty. Check that your tests ran before the upload.
- Nothing arrives when tests fail. Run the upload even after a failed step: if: always() on GitHub Actions, after_script on GitLab CI.
- Tests split into two histories. The automation ID uses the file path and suite. If your reporter's paths change between machines, the IDs change too. Run from the same working directory each time.
Frequently asked questions
What does CTRF stand for?
Common Test Report Format. It is an open standard for JSON test reports, described at ctrf.io.
Does TestPlum detect CTRF on its own?
Yes, the upload endpoint detects CTRF, Allure and JUnit XML from the content. The command on this page adds format=ctrf anyway, so the intent is clear.
Are CTRF retries shown?
Yes. Each retry attempt becomes its own result, in order, before the final one. A pass on retry counts for less in the trusted pass rate.
What happens to the CTRF status other?
It is recorded as blocked. Pending is recorded as skipped.
Can I send JUnit XML and CTRF to the same project?
Yes. Both go to the same endpoint. Their automation IDs are built differently, so the same test sent in both formats would have two histories. Pick one format per suite.
Does TestPlum make its own CTRF reporter?
No. Use the CTRF reporter for your test tool from the CTRF project or its community.
Sources
- CTRF: Common Test Report Format, checked November 19, 2026
- CTRF specification: status values, checked November 19, 2026
CTRF and the test tools named on this page are trademarks of their owners. TestPlum is not affiliated with or endorsed by them.