GitLab CI integration
GitLab CI test results, with a release answer that can stop the pipeline
Two lines in after_script send your results and print the answer. One small job at the end of the pipeline can stop it when the release is not ready.
Add an after_script to the GitLab CI job that runs your tests: it uploads your JUnit XML to TestPlum with a masked CI key, even when tests fail. To gate the pipeline, add a job in the .post stage with when: always that fails unless the answer is Ready.
Step 1: create a CI key and save it as a variable
- Open Connect CI. In TestPlum, go to Runs → Connect CI. Pick your test tool and GitLab CI.
- Create the key. Press Create CI key. It is shown once.
- Save it in GitLab. In your project: Settings → CI/CD → Variables → Add variable. Name it TESTPLUM_CI_KEY and tick Mask variable.
Step 2: add the upload and the gate to .gitlab-ci.yml
The first part goes inside the job that runs your tests. The second part is a new job. Change junit.xml to your report's path; the Playwright setup writes results.xml. Connect CI fills in your project ID.
If you only want to see the answer, not gate on it, choose Show the answer in the build log on Connect CI instead. You then get just the after_script, with the answer printed as its second line.
# In the job that runs your tests:
after_script:
- 'curl -f -sS -X POST -H "Authorization: Bearer $TESTPLUM_CI_KEY" -H "Content-Type: application/xml" --data-binary @junit.xml -o /dev/null "https://app.testplum.com/api/v1/projects/<project-id>/automation/report?release=%40next&run_name=CI%20%23$CI_PIPELINE_IID"'
# A new job, run last, that stops the pipeline unless the answer is Ready:
testplum-answer:
stage: .post
when: always
image:
name: curlimages/curl
entrypoint: [""]
script:
- 'curl -f -sS -H "Authorization: Bearer $TESTPLUM_CI_KEY" "https://app.testplum.com/api/v1/projects/<project-id>/status?format=text" | tee testplum-answer.txt'
- 'head -n 1 testplum-answer.txt | grep -q ''^Ready'' || { echo "TestPlum: not ready to ship, so the build stops here."; exit 1; }'Why after_script and a .post job
GitLab runs after_script even when the script section fails, so your failures are uploaded too. But GitLab's docs are clear that after_script does not change the job's exit code. A check there could never fail the job.
So the gate lives in its own job. The .post stage runs after your other stages by default, and when: always runs the job whatever happened in earlier stages. It uses the small curlimages/curl image, asks TestPlum for the answer and fails unless the first line starts with Ready. A missing answer fails it too.
What arrives in TestPlum
Each pipeline becomes a run named CI # and the pipeline's IID. It counts toward release=@next: the open release due soonest, the one the project home answers for.
- Run history. Each upload becomes a run named after the build, such as CI #412, next to your manual runs.
- Flaky detection and quarantine. Tests whose results flip between pass and fail are suggested for quarantine. A quarantined test is left out of the verdict counts until it passes enough times in a row, or the quarantine expires.
- Failure grouping by cause. Failures with the same normalized error become one group, sorted as product bug, broken test, environment or flaky, with the reasons shown.
- Trust-weighted pass rate. Each pass counts by how much it proves. A pass on retry, a flaky test or a test edited after the run counts for less, so a 100% pass rate can still fail the trusted pass rate check.
- A release verdict in plain words. The project answers Ready or Not ready, with each reason in one line and a link to where to fix it.
- Answer history. TestPlum remembers when the answer turned and what changed: up to five changes, newest first, with the reasons that were new or fixed.
Security: a CI key locked to one project
The key you paste into CI is a CI key, made under Runs → Connect CI. It belongs to one project and can do three things there: send results, run the release gate and read the status. Any other request with it is refused with 403, and a revoked key gets 401.
A CI key acts as the person who created it, so it can never do more than they can. Only a signed-in person who can run and edit tests can create one. It lasts a year, a project can have at most 20 live keys, and creating or revoking a key is written to the audit log.
Store it as a masked secret in your CI, never in a file you commit. If it leaks, revoke it on the same Connect CI page and make a new one.
Troubleshooting
- The gate job never runs. GitLab does not run a pipeline that only has .pre or .post jobs. It needs your normal test job too.
- The .post job cannot find the key. It reads the same TESTPLUM_CI_KEY variable. Check that the variable is not limited to an environment or protected branches the job does not run on.
- The upload ran, but the job still passed with failed tests. That is expected: the job passes or fails on your test script. after_script only reports.
- The YAML will not parse. Each command is a single-quoted string, and a quote inside it is written twice, as in ''^Ready''. Paste the snippet as it is.
- 401 from TestPlum. The key was revoked or copied with a missing character. Make a new one on Connect CI.
Frequently asked questions
Why does the gate need its own job?
Because after_script cannot fail a GitLab job. The .post job runs last and fails the pipeline unless the answer is Ready.
Does the upload happen when tests fail?
Yes. GitLab runs after_script after a failed script, so failed tests are sent too.
Does TestPlum use GitLab's test report feature?
No, the two are separate. You can keep artifacts:reports:junit for GitLab's own view and also send the same file to TestPlum.
Which runner image does the gate job use?
curlimages/curl, with an empty entrypoint so GitLab can run the script. Any image with curl, grep and head works.
Can I use GitLab's built-in variables in the run name?
The snippet already does: $CI_PIPELINE_IID names each run after its pipeline.
Can several projects share one key?
No. A CI key is locked to one TestPlum project. Make one key per project.
Sources
- GitLab Docs: CI/CD YAML syntax (after_script, stages and .post, when: always), checked November 19, 2026
GitLab is a trademark of its owner. TestPlum is not affiliated with or endorsed by it.