Playwright integration
Playwright test reporting with a results dashboard and a release verdict
Playwright already writes a JUnit report. Send that file to TestPlum from CI and every run lands in one history, next to your manual tests, with an answer to the question that matters: can we ship?
Add Playwright's built-in JUnit reporter to playwright.config, then send results.xml to TestPlum with one curl step and a project CI key from Runs → Connect CI. TestPlum keeps the run history, groups failures by cause, flags flaky tests and says whether the release is Ready or Not ready.
Step 1: have Playwright write a JUnit file
Playwright ships a JUnit reporter, and you can run several reporters at once, so you keep the output you see today. Add this line to the config in playwright.config:
reporter: [["list"], ["junit", { outputFile: "results.xml" }]],Step 2: send the file after your tests
In TestPlum, open Runs → Connect CI, pick Playwright and your CI, and press Create CI key. Save the key as a secret named TESTPLUM_CI_KEY. Then add this after the step that runs your tests. Replace <project-id> with your project's ID; the Connect CI page fills it in for you.
The first command sends results.xml. The second asks for the answer and prints it in the build log. For a ready-made step, see the GitHub Actions and GitLab CI pages.
# After your tests, with TESTPLUM_CI_KEY set to the key:
curl -f -sS -X POST \
-H "Authorization: Bearer $TESTPLUM_CI_KEY" \
-H "Content-Type: application/xml" \
--data-binary @results.xml -o /dev/null \
"https://app.testplum.com/api/v1/projects/<project-id>/automation/report?release=%40next&run_name=CI%20%23$BUILD_NUMBER"
curl -f -sS -H "Authorization: Bearer $TESTPLUM_CI_KEY" "https://app.testplum.com/api/v1/projects/<project-id>/status?format=text" | tee testplum-answer.txtWhat you get in TestPlum
Connect CI waits for your first results and shows them the moment they arrive, with the counts and a link to the answer. From then on, every pipeline adds a run.
- Run history. Each upload becomes a run named after the build, such as CI #412, next to your manual runs.
- Flaky detection and quarantine. Tests whose results flip between pass and fail are suggested for quarantine. A quarantined test is left out of the verdict counts until it passes enough times in a row, or the quarantine expires.
- Failure grouping by cause. Failures with the same normalized error become one group, sorted as product bug, broken test, environment or flaky, with the reasons shown.
- Trust-weighted pass rate. Each pass counts by how much it proves. A pass on retry, a flaky test or a test edited after the run counts for less, so a 100% pass rate can still fail the trusted pass rate check.
- A release verdict in plain words. The project answers Ready or Not ready, with each reason in one line and a link to where to fix it.
- Answer history. TestPlum remembers when the answer turned and what changed: up to five changes, newest first, with the reasons that were new or fixed.
How Playwright results match your tests
Each test case in the JUnit file gets an automation ID built from its class name and test name. The next upload with the same ID adds a result to the same TestPlum test, so its history grows run after run.
A test TestPlum has not seen yet is created for you in an Automated suite, under a section named after its test suite. Failures and errors both count as failed, skipped tests as skipped, and the failure message is kept on the result.
Rename a test and it gets a new automation ID, so it starts a new history. If you want to keep the old one, edit the automation ID on the existing test in TestPlum.
Stop the build when the release is Not ready
On Connect CI you can choose Stop the build when Not ready. The step then ends with one more line. It reads the first line of the answer and fails the build unless it starts with Ready. A missing answer stops the build too.
head -n 1 testplum-answer.txt | grep -q '^Ready' || { echo "TestPlum: not ready to ship, so the build stops here."; exit 1; }Security: a CI key locked to one project
The key you paste into CI is a CI key, made under Runs → Connect CI. It belongs to one project and can do three things there: send results, run the release gate and read the status. Any other request with it is refused with 403, and a revoked key gets 401.
A CI key acts as the person who created it, so it can never do more than they can. Only a signed-in person who can run and edit tests can create one. It lasts a year, a project can have at most 20 live keys, and creating or revoking a key is written to the audit log.
Store it as a masked secret in your CI, never in a file you commit. If it leaks, revoke it on the same Connect CI page and make a new one.
Troubleshooting
- Nothing arrives when tests fail. Most CI systems skip later steps after a failure. Make the upload step run anyway: if: always() on GitHub Actions, or after_script on GitLab CI.
- curl says the file is missing. The path in --data-binary @results.xml is relative to where the step runs. Match it to outputFile in playwright.config.
- 403 from TestPlum. The key belongs to another project, or the URL has the wrong project ID.
- The run counts toward no release. release=@next means the open release due soonest. With no open release, the run is tied to none. Create a release in TestPlum and the next upload counts toward it.
- Retries do not show as separate attempts. A JUnit file holds one outcome per test. Flaky tests still show up across runs. If you need each retry recorded, send a CTRF report instead, which carries retry attempts.
Frequently asked questions
Do I need to change my Playwright tests?
No. You add one reporter line to playwright.config and one step to your pipeline. The tests themselves stay as they are.
Is there a TestPlum Playwright reporter on npm?
Not yet. TestPlum has its own reporter that records every attempt with its steps, but it is not published as a public package. Playwright's built-in JUnit reporter works today with no extra install.
Will my existing reporters keep working?
Yes. Playwright runs every reporter listed in the config, so the list or HTML reporter you use now keeps working next to JUnit.
Do failed tests still get uploaded?
Yes, as long as the upload step runs after a failure. Use if: always() on GitHub Actions or after_script on GitLab CI.
Where do I find my project ID?
Open Runs → Connect CI in your project. The step it shows already contains the project ID and the right address.
Can the pipeline fail when the release is not ready?
Yes. Choose Stop the build when Not ready on Connect CI. The step then exits with an error unless the answer's first line starts with Ready.
Sources
- Playwright: Reporters (JUnit reporter, multiple reporters), checked November 19, 2026
- GitHub Docs: Expressions, always(), checked November 19, 2026
- GitLab Docs: CI/CD YAML syntax, after_script, checked November 19, 2026
Playwright is a trademark of its owner. TestPlum is not affiliated with or endorsed by it.