GitHub Actions integration

GitHub Actions test results, with the release answer in every run

Add one step to your workflow. It sends your test results to TestPlum, prints whether the release is ready in the build log and the job summary, and can stop the build when it is not.

By the TestPlum teamChecked

Short answer

Paste one step after your tests in a GitHub Actions workflow, with a TestPlum CI key saved as a repository secret. The step uploads your JUnit XML, writes the Ready or Not ready answer to the job summary, and can fail the job unless the release is Ready.

Step 1: create a CI key and save it as a secret

  1. Open Connect CI. In TestPlum, go to Runs → Connect CI. Pick your test tool and GitHub Actions.
  2. Create the key. Press Create CI key. It is shown once, so copy it now.
  3. Save it in GitHub. In your repository: Settings → Secrets and variables → Actions → New repository secret. Name it TESTPLUM_CI_KEY.

Step 2: add the step to your workflow

Put this under steps: in the job that runs your tests, after the test step. It is indented for that place. Change junit.xml to the file your test tool writes; the Playwright setup writes results.xml. Connect CI fills in your project ID.

.github/workflows/your-workflow.yml
      # After the step that runs your tests:
      - name: Send results to TestPlum
        if: always()
        env:
          TESTPLUM_CI_KEY: ${{ secrets.TESTPLUM_CI_KEY }}
        run: |
          curl -f -sS -X POST \
            -H "Authorization: Bearer $TESTPLUM_CI_KEY" \
            -H "Content-Type: application/xml" \
            --data-binary @junit.xml -o /dev/null \
            "https://app.testplum.com/api/v1/projects/<project-id>/automation/report?release=%40next&run_name=CI%20%23${{ github.run_number }}"
          curl -f -sS -H "Authorization: Bearer $TESTPLUM_CI_KEY" "https://app.testplum.com/api/v1/projects/<project-id>/status?format=text" | tee testplum-answer.txt
          sed '1s/^/### /' testplum-answer.txt >> "$GITHUB_STEP_SUMMARY"

What the step does

  • if: always() makes the step run even when the test step failed, so failures are reported too.
  • The first curl sends the report. The run is named CI # and the workflow's run number, and counts toward release=@next: the open release due soonest.
  • The second curl asks for the answer as plain lines and prints it. The first line says Ready only when the release is ready. Then come the release, the numbered reasons and a link.
  • The sed line adds the answer to the job summary, with the first line as a heading, so you see it on the workflow run page without opening the log.

Stop the build when Not ready

To make the release answer a gate, choose Stop the build when Not ready on Connect CI. The step gets one more line at the end of run:. It fails the job unless the answer starts with Ready. If TestPlum could not answer, the job fails too, so a missing answer never passes by accident.

Last line of the run: block
head -n 1 testplum-answer.txt | grep -q '^Ready' || { echo "TestPlum: not ready to ship, so the build stops here."; exit 1; }

What you see in TestPlum

  • Run history. Each upload becomes a run named after the build, such as CI #412, next to your manual runs.
  • Flaky detection and quarantine. Tests whose results flip between pass and fail are suggested for quarantine. A quarantined test is left out of the verdict counts until it passes enough times in a row, or the quarantine expires.
  • Failure grouping by cause. Failures with the same normalized error become one group, sorted as product bug, broken test, environment or flaky, with the reasons shown.
  • Trust-weighted pass rate. Each pass counts by how much it proves. A pass on retry, a flaky test or a test edited after the run counts for less, so a 100% pass rate can still fail the trusted pass rate check.
  • A release verdict in plain words. The project answers Ready or Not ready, with each reason in one line and a link to where to fix it.
  • Answer history. TestPlum remembers when the answer turned and what changed: up to five changes, newest first, with the reasons that were new or fixed.

Security: a CI key locked to one project

The key you paste into CI is a CI key, made under Runs → Connect CI. It belongs to one project and can do three things there: send results, run the release gate and read the status. Any other request with it is refused with 403, and a revoked key gets 401.

A CI key acts as the person who created it, so it can never do more than they can. Only a signed-in person who can run and edit tests can create one. It lasts a year, a project can have at most 20 live keys, and creating or revoking a key is written to the audit log.

Store it as a masked secret in your CI, never in a file you commit. If it leaks, revoke it on the same Connect CI page and make a new one.

Troubleshooting

  • The step is skipped when tests fail. Check that if: always() is on the TestPlum step itself, not only on the job.
  • You do not want it to run on a cancelled workflow. GitHub's docs note that always() runs even when the job is cancelled, and suggest if: ${{ !cancelled() }} for steps that should run on success or failure only. Either works with TestPlum.
  • curl: (22) The requested URL returned error: 401. The key was revoked or mistyped. Make a new one on Connect CI and update the secret.
  • File not found. The path after --data-binary @ is relative to the working directory of the step. Match it to your reporter's output file.

Frequently asked questions

Is there a TestPlum GitHub Action on the Marketplace?

Not yet. The step on this page uses curl, which is already on GitHub's hosted runners, so there is nothing to install.

Does TestPlum post commit statuses or comments to GitHub?

Not from this step. It prints the answer in the log and the job summary, and can fail the job. Your branch protection can then require that job.

Which test tools work?

Any tool that writes JUnit XML, CTRF or Allure. Connect CI has ready setups for Playwright, Jest, Vitest and pytest.

Will results count toward the right release?

Yes, with release=@next they count toward the open release due soonest, the one the project home answers for. You never edit the step between releases.

Can I see the answer without failing the build?

Yes. That is the default, Show the answer in the build log. The job passes or fails on your tests, as before.

What if TestPlum is unreachable?

curl -f fails the step. With the stop line, a missing answer also stops the build, so decide whether that suits your pipeline.

Sources

GitHub is a trademark of its owner. TestPlum is not affiliated with or endorsed by it.

Find out whether your next release is ready

Create a workspace, import your tests or write a few, and record a run. Everything on this site is in the product today.