The TestPlum user guide
How to manage the testing of a large product in TestPlum, from sign-up to shipping a release, in plain language with one worked example throughout.
Create a workspace, invite your team with the right roles, make one project per product that ships on its own, group tests in folders by feature, import or write tests, link them to requirements, plan each release and sprint as runs, record results by hand and from CI, sort failures by cause, and ship when the project home shows Ready. This guide walks through every step with an enterprise example.
What TestPlum does, in one minute
TestPlum keeps all your testing in one place and turns it into one answer: is this release ready to ship, or not?
It works in four steps, and the whole product is built around them:
Tests ──► Results ──► Analysis ──► Ready or Not ready
what to people and failures a green or red light,
check CI record sorted by with each reason in
results cause one plain sentence
- Tests: you write them, import them from another tool, or let your automated tests create them.
- Results: testers record results by hand, and your CI pipeline sends automated results.
- Analysis: TestPlum groups failures by cause, finds flaky tests and notices passes that prove little.
- The answer: open a project and the first thing you see is a green or red light for the release you are working on, with each reason written as a sentence such as “Only 86.7% of tests pass. Needs 95%.” and a button that takes you to where you fix it.
The same answer shows up everywhere: on the project home, in your build log, in Slack or Teams, in an email, in a README badge and in AI coding assistants.
Words you will see
| Word | What it means | Example |
|---|---|---|
| Workspace (organization) | Your company’s space in TestPlum. Everything lives inside it. | Northwind |
| Project | One product or app you test. Has its own tests, runs, bugs and releases. | Northwind Commerce |
| Key | A short code for a project. Every item gets a key built from it. | Project NWC, test NWC-C42, bug NWC-BUG7 |
| Folder / subfolder | How tests are grouped, usually by feature. | Checkout › Payment |
| Test | One thing to check, with steps and expected results. | Pay with a saved card |
| Step | One action in a test and what should happen. | Click Pay now → An order number is shown |
| Requirement | Something the product must do, often a user story. Tests are linked to it. | Customers can save a card (NWC-R12) |
| Run | A set of tests to execute, with a result for each one. | Sprint 24 regression (NWC-RUN31) |
| Result | What happened when a test ran. | Passed, Failed, Blocked, Skipped, Rerun, Not run |
| Cycle | A reusable plan for runs you repeat, such as a weekly regression. | Weekly regression |
| Release | A version you plan to ship, with its own runs and rules for “ready”. | 5.2 (build 5.2.0-rc1, due 30 May) |
| Bug | A problem found while testing. Can be sent to Jira or GitHub. | NWC-BUG7: Saved card not preselected |
| The answer (verdict) | Ready or Not ready, with the reasons. | Not ready: 2 things to fix |
| Check | One rule that decides the answer, with a target. | Pass rate ≥ 95% |
| Trusted pass rate | The pass rate, counting each pass by how much it really proves. | 80%, because some passes are weak |
| Flaky test | A test that passes and fails without any change. | Avoids an obstacle |
| Quarantine | Taking a flaky test out of the answer for a while, until it is fixed. | 14 days, 10 passes in a row to confirm a fix |
| CI key | A secret your pipeline uses to send results. Works for one project only. | Shown once, saved as TESTPLUM_CI_KEY |
The example used in this guide
Throughout this guide we test Northwind Commerce, a made-up enterprise e-commerce platform:
- Parts of the product: web shop, checkout and payments, product catalog and search, customer accounts, admin console, iOS and Android apps, and a public REST API.
- People: 3 squads (Checkout, Catalog, Platform), 8 QA engineers, 25 developers, 2 product managers and a release manager.
- How they work: two-week sprints. A release every month (5.1, 5.2, 5.3…), plus hotfixes. A full regression before every release. Automated tests (Playwright for web, pytest for the API) run in GitHub Actions on every pull request and every night.
- Today: about 2,500 manual tests in TestRail and spreadsheets, and user stories in Jira.
You can copy this setup and change the names to match your product.
Plan your setup before you click anything
Ten minutes of planning saves weeks of reorganising. Answer these five questions first.
Question 1: How many projects?
Rule of thumb: one project per product that is released on its own schedule. A project has one answer, so put things together that ship together.
| Your situation | What to do |
|---|---|
| Web, mobile and API ship together as one version | One project, with folders for each part |
| Mobile apps ship on their own schedule (App Store reviews) | Two projects: Northwind Commerce and Northwind Mobile |
| A separate product with its own team and releases | Its own project |
For Northwind, web and API ship together, and the mobile apps ship separately:
Northwind (workspace)
├── Northwind Commerce key NWC web shop, checkout, catalog, accounts, admin, API
└── Northwind Mobile key NWM iOS and Android apps
On the Free plan you can have 1 active project and 3 testers. An enterprise product will need the Team or Business plan. See Plans.
Question 2: How do we group tests?
By feature area, then by sub-feature. Not by sprint, not by tester, and not by “manual vs automated”. Sprints and people change; features stay. You will use tags and runs for sprints.
Checkout
├── Cart
├── Payment
├── Discounts
└── Guest checkout
Catalog
├── Search
└── Product page
Accounts
├── Sign up and sign in
└── Saved cards
Admin console
├── Orders
└── Refunds
Public API
├── Orders API
└── Webhooks
Question 3: What does a release look like?
In TestPlum, a release is the thing you decide to ship (5.2), and runs are the work of testing it (several per sprint). A release can contain runs from more than one sprint.
Release 5.2 (due 30 May)
├── Sprint 23 – Checkout new features (run)
├── Sprint 24 – Checkout and Catalog (run)
├── Release 5.2 full regression (run, from the "Full regression" cycle)
└── CI #418, CI #419, … (automated runs from GitHub Actions)
Question 4: Who needs what access?
Write down who will write tests, who will run tests, who will only look, and who manages the project. People who only look are free on every plan. See Step 2.
Question 5: What does “ready” mean for you?
Agree on the rules before the first release, not the night before shipping. The defaults are a good start:
| Check | Default target | Meaning |
|---|---|---|
| Runs in scope | at least 1 | Something was tested for this release |
| Execution completion | 100% | Every test in scope has a final result |
| Pass rate | 95% | Passed ÷ tests that were run (skipped tests are not counted) |
| Trusted pass rate | 90% | The pass rate when weak passes count for less |
| Critical tests failing or blocked | 0 | No critical-priority test is failing or blocked |
| Open critical bugs | 0 | No open bug with critical priority or blocker/critical severity |
| Requirements verified | 80% | Requirements proven by passing tests |
| Quarantined critical tests | 0 | No critical test is hidden in quarantine |
You can change the targets for each release.
Step 1: Create your workspace
- Go to the TestPlum sign-up page and fill in Your name, Work email, a Password (at least 10 characters) and your Organization name (for example Northwind).
- Click Create account. You land on Projects with a short welcome guide.
You are now the owner of the workspace.
Tips
- Press ⌘K (Mac) or Ctrl+K (Windows) anywhere to search or jump to any test, run, bug or requirement by name or key.
- Open Account (bottom left) to set your time zone, so dates and deadlines show in your local time.
- Want to look around first? Click Try with sample data on the Projects page. You get a Sample: Web Shop project filled with tests, requirements, a release with results and a bug, so you can see a real answer before entering anything. It does not count toward your plan, and you can delete it any time. Its home has a short guide, Make 1.0 Ready yourself: retest what failed, give the uncovered requirement a test, finish the run, and watch the light turn green. Each step ticks itself as you go.
- The screen icon at the top right switches the look between System, Light and Dark. TestPlum also follows your computer’s reduce transparency and increase contrast settings.
Step 2: Invite your team and give everyone the right access
Organization roles
| Role | What they can do |
|---|---|
| Owner | Everything, including the plan and other owners. There is always at least one owner. |
| Admin | Sees every project and all settings. |
| Member | Sees only the projects they are added to, with the role you give them there. |
Project roles
Inside each project, a member gets one of four roles. You can edit these roles or create your own under Organization settings → People and access → Roles.
| What you can do | Viewer | Tester | Lead | Manager |
|---|---|---|---|---|
| See tests, requirements, runs, bugs and reports | ✅ | ✅ | ✅ | ✅ |
| Record results, report and update bugs | ✅ | ✅ | ✅ | |
| Write, edit and delete tests and requirements | ✅ | ✅ | ||
| Create, edit and delete runs | ✅ | ✅ | ||
| Manage the project (settings, alerts, badge, CI) | ✅ | |||
| Sign releases (electronic signatures) | ✅ |
Northwind example
| Who | Organization role | Project role |
|---|---|---|
| QA manager | Admin | (sees everything) |
| QA leads, one per squad | Member | Lead in Northwind Commerce |
| QA engineers | Member | Tester |
| Release manager | Member | Manager |
| Developers who fix bugs | Member | Tester (they can update bugs and record results) |
| Product managers, support, executives | Member | Viewer (free) |
How to invite people
- Go to Organization settings → People and access → Members and click Invite people.
- Enter one or more email addresses, separated by commas or new lines (up to 50 at a time).
- Pick the Organization role. For a member, use Add a project… to give access to each project, with a role.
- Click Send invitations.
Each invitation works once and lasts 7 days. Project access starts when the person accepts.
Invitation did not arrive? Ask them to check their spam folder. An admin can also open Pending invitations, click Copy new link and send the link another way, such as in chat. The old link stops working.
Changing access later
- Change someone’s role in the Members table, or their project role under Project settings → General → Team.
- Set a person to Disabled to sign them out everywhere and block sign-in. Their work and history stay.
- Remove takes them out of the organization.
Step 3: Create your projects
- On Projects, click New project.
- Enter a Name (Northwind Commerce). TestPlum suggests a Key (NC); change it to something short you
like (NWC, 2 to 10 capital letters or digits). The key appears in every item, such as
NWC-C42, so keep it short. - Add a Description and click Create project.
The project opens on its home, with a grey light and a three-step guide: Add your tests, Record results, Get your answer.
Moving around a project
The left sidebar has everything:
Home the answer, what needs you, the four steps
Tests all tests in folders
Requirements user stories and coverage
Maintenance duplicates and stale tests
Runs test runs
Cycles reusable run plans
Sessions exploratory testing
Automation runners, schedules and test selection
Connect CI send automated results
Failures failures sorted by cause
Flaky tests flaky tests and quarantine
Bugs bugs
Fixes proposed fixes for broken automated tests
Releases releases and their answers
Insights what to test next, stability, workload, custom reports
Activity who did what
Project settings (managers only)
Project settings (managers)
- General: name, key, description; Team (who has access and their role); Test templates; and a Danger zone to Copy this project, Archive project (archived projects are read-only and hidden from the default list) or Delete project.
- Notifications: Slack and Microsoft Teams channels, the README Status badge, and Share the answer.
- Activity: the project’s history.
Step 4: Set up how you describe tests
Do this once for the whole workspace, under Organization settings → Workflow. It keeps every team describing tests the same way.
| Tab | What it is | Northwind example |
|---|---|---|
| Fields | Extra fields on tests. Types: text, long text, number, checkbox, date, dropdown, multi-select, link, person. | Platform (multi-select: Web, iOS, Android, API), Squad (dropdown), Jira epic (link) |
| Field layouts | Which extra fields appear for which kind of test. | Mobile test layout shows Device and OS version |
| Result names | The names of results. Defaults: Not run, Passed, Failed, Blocked, Skipped, Rerun. | Keep the defaults |
| Test types | Kinds of tests. Defaults: Feature, Regression, Smoke, Edge case, Exploratory, Non-functional. | Add Security, Accessibility |
Priorities are fixed: Critical, High, Medium, Low. Use them carefully, because critical tests have special rules: one failing or blocked critical test makes the answer Not ready. Mark a test Critical only if you would stop a release when it fails (paying, signing in, placing an order).
Test templates (Project settings → General → Test templates): save a test as a template (Save as template on any test) so people can start new tests with common steps already filled in, such as “Sign in as a customer”.
Step 5: Organise tests into folders by feature
On Tests, the left panel shows your folders.
- Click + Folder to add a top-level folder (Checkout). Add subfolders inside it (Payment, Cart). Subfolders can go as deep as you need.
- A folder’s number includes everything in its subfolders. Clicking a folder lists all of its tests, including subfolders.
- Rename or delete a folder from its menu (the ⋯ next to it).
Find tests fast
- Type in Search tests… to match a title or a key (
NWC-C42). - Filter by priority, tag, type and more.
- Save a filter you use often with Saved views → + Save current filters…, for example “Checkout critical tests”.
- Switch between Comfortable and Compact rows.
- Select several tests to move, re-prioritise, tag or delete them in one go.
- Export CSV downloads the tests (with your filters).
Tags are for things that cut across folders: smoke, regression, checkout-squad, pci, sprint-24. Use
tags to build runs, for example “every test tagged smoke”.
Step 6: Write good tests
Click New test (or Write a test on a new project’s home).
| Field | What to put | Example |
|---|---|---|
| Title | What is checked, as a short sentence | Pay with a saved card |
| Folder / Subfolder | Where it belongs | Checkout › Payment |
| Priority | Critical, High, Medium or Low | Critical |
| Type | Feature, Regression, Smoke… | Regression |
| Owner | Who looks after this test | Priya (Checkout squad) |
| Estimate (minutes) | How long it takes by hand | 5 |
| Tags | Comma separated | smoke, pci |
| References | Links to tickets, optionally Label=ID |
JIRA=NW-1234 |
| Preconditions | What must be true before you start | Signed in as a customer with a saved Visa card. The cart holds one item. |
| Steps | Each action and what should happen | see below |
Example steps
| # | Action | Expected result |
|---|---|---|
| 1 | Open the cart and select Checkout | The checkout page opens with the saved card preselected |
| 2 | Select Pay now | The confirmation page shows an order number and the total charged |
| 3 | Open Orders from the account menu | The new order is listed with the status Paid |
Click + Add step for each step (or press ⌘/Ctrl + Enter while typing a step). Click Create test, or Create & add another to keep going.
What makes a good test
- One idea per test. “Pay with a saved card” and “Declined card shows an error” are two tests.
- Every step has an expected result. A manual pass without an expected result counts for less in the trusted pass rate, because nobody knows what was checked.
- Write for someone new. Name the exact button and the exact message.
- Put setup in Preconditions, not in step 1.
- Formatting: preconditions and notes support simple Markdown (
**bold**, lists, links), with a Preview button.
After you save
A test page has tabs:
- Details: everything above.
- Executions: every result this test ever had, in every run.
- Versions: every save makes a new version. Compare any two. Runs remember the version they ran, so old results always match the steps that were really executed.
- Comments: discuss the test. Type @ to mention someone; they get a notification.
- Files: screenshots and documents. You can also paste images straight into a text field.
Buttons on a test: Edit, Duplicate, Save as template, Automate (makes a Playwright test file from the steps, see For developers) and Follow (get notified when it is commented on).
AI help (optional). If AI is switched on for your workspace, TestPlum can draft tests from a requirement for a person to review. AI-written tests count only partly in the answer until someone reviews them.
Step 7: Bring in tests you already have
On Tests, click Import and choose where your tests come from.
| Source | What to export there | What comes across |
|---|---|---|
| TestRail | An XML (best) or CSV export of a suite | Sections, steps, preconditions, priority, type, estimate, references; other fields go to matching custom fields or the description |
| Xray | Bulk JSON, GraphQL results or CSV | Folders, steps with test data, labels, priorities, requirement links |
| Zephyr Scale | XML or CSV export of test cases | Folders, steps and test data, objective, precondition, priority, labels, covered issues, custom fields |
| CSV file | Any spreadsheet saved as CSV | See the columns below |
How every import works
- You choose the file. TestPlum checks it first and shows what will happen: new, updated, unchanged and skipped tests, and anything it cannot carry over.
- Nothing is saved unless every test is valid. Errors name the row.
- Each test keeps its original ID. Importing a newer export later updates the same tests instead of adding copies. You can keep both tools side by side for a while.
- Big imports continue in the background. A progress tray follows you on every page, and you can cancel.
After a TestRail or Xray import, you can also copy each test’s attachments and pasted images from the old tool.
Past results: under Import → Past results you can bring in old TestRail run results or Xray results. They become signed-off runs with their dates and testers. They never change the current answer.
Importing from a spreadsheet (CSV)
Save your spreadsheet as CSV (UTF-8) with a header row. One row per step: a test’s first row has its fields and its first step; each further step is a row with only Step and Expected filled.
Columns TestPlum reads (names are not case-sensitive):
| Column | Meaning |
|---|---|
| Title (required) | The test title |
| Folder, Subfolder (or Section) | Where it goes. Nest with >: Checkout > Payment |
| Priority | Critical, High, Medium, Low |
| Type, Tags, Estimate (min) | As in the editor |
| Preconditions, Description, Notes | Text (Notes is added to the description) |
| Step, Expected (or Expected result) | One step per row |
| ID or External ID | Your own ID. Kept, so importing the same file again adds nothing twice |
| Automation ID, Automated | Links the test to its automated results |
| Requirement | Links requirements that already exist (by key such as NWC-R12, or by external ID) |
| Custom fields | By their name, or as cf:key |
Example CSV
ID,Title,Section,Priority,Preconditions,Step,Expected result,Requirement
CHK-1,Pay with a saved card,Checkout > Payment,Critical,Signed in with a saved card,Open the cart and select Checkout,The saved card is preselected,REQ-12
,,,,,Select Pay now,An order number and the total are shown,
CHK-2,Declined card shows an error,Checkout > Payment,High,Signed in,Pay with a card that is always declined,"""Your card was declined"" is shown",REQ-13
The check says, for example, “2 tests ready to import (3 rows: a test’s further steps take rows of their own)”. It also names any column it does not read and lists requirements it could not find.
Step 8: Add requirements and see coverage
Requirements are what the product must do: user stories, epics or regulations. Linking tests to them shows what is proven to work.
Adding requirements
Go to Tests → Requirements:
- New requirement: a Title, an External ID (such as the Jira key NW-1234), Tags, a Description, Acceptance criteria (one per line) and the tests that verify it (tick them).
- Import from Jira: bring in stories with a JQL search, for example
project = NW AND sprint = "Sprint 24". They are keyed by their Jira key, follow Jira’s title, description and status, and TestPlum keeps one comment on each story listing the tests that cover it.
What the statuses mean
Each requirement gets a status from the latest results of its linked tests:
| Status | Meaning |
|---|---|
| Not covered | No tests are linked |
| Not run | Tests are linked, but none has a result yet |
| Failing | At least one linked test failed or is blocked |
| Partial | Some linked tests passed, others have not run yet |
| Passing | All linked tests passed |
A requirement that passes only on weak evidence (for example, only on tests with no assertions) does not count as verified.
Coverage tools
- Traceability matrix: requirements down the side, tests across, results in the cells. Good for audits.
- Coverage gaps (on a requirement): what is not covered yet.
- Write a test for it (on a requirement with no tests): writes a new test and links it to the requirement in one step.
- Filter requirements by status, and save views.
Northwind example: at sprint planning, the QA lead imports the sprint’s stories from Jira, then links or writes tests for each one. A story with no tests shows as Not covered, which is the reminder to write them.
Step 9: Plan a release and a sprint
Create the release
Go to Releases → New release:
| Field | Example |
|---|---|
| Name | 5.2 |
| Build | 5.2.0-rc1 |
| Environment | Staging · Chrome 130 · iOS 18 |
| Target date | 30 May |
| Thresholds for this release | Min pass rate 95%, min trusted pass 90%, min executed 100%, min requirements verified 80%, max open critical bugs 0 |
| Required signatures (Business plan) | Reviewed by QA, Approved for release |
Why releases matter so much: the project home answers for the open release due soonest. If no release is open, it answers for all open runs. Always keep a release open for what you are working toward, so the answer is about the right thing.
On the release page you can Choose runs (add existing runs), start a New run for it, and narrow down which requirements count for this release.
Create a cycle for work you repeat
Go to Runs → Cycles → New cycle. A cycle is a reusable run plan, such as Weekly regression (every test tagged
regression) or Release full regression (everything). Each time, click Start run from cycle and you get a fresh
run with the same selection.
Create a run
Go to Runs → New run (or Run tests on the home):
| Field | Example |
|---|---|
| Name | Sprint 24 – Checkout |
| Environment | Staging · Chrome 130 |
| Release | 5.2 |
| Deadline | Fri 24 May (adds an ideal line to the burndown chart) |
| Cycle | (none) or Weekly regression |
| Run owner | Priya |
| Assign all tests to | (unassigned), or one person |
| Tests | see below |
Choosing the tests
| Option | What you get | When to use it |
|---|---|---|
| Suggested (default) | Tests that failed or were blocked last time, never ran, changed since they ran, or are critical. It tells you how many of each. | Most sprint runs |
| Everything | Every test in the project | A full regression |
| By filter | Tests matching folder, priority, tag, type or last result | Every Checkout test tagged smoke |
| Hand-picked | Tests you tick one by one. The suggested tests start ticked; use Clear to start empty. | A focused retest |
A run takes a snapshot of each test. If someone edits a test later, the run still shows the steps it was created with, and marks the test as changed.
Forgot something? Click Add tests on the run to add more by any of the same options. Tests already in the run keep their results.
Step 10: Run tests by hand
Open the run. You see:
- The barcode: one segment per test, coloured by result, with “27 of 39 executed (69.2%) · 22 passed · 2 failed · 3 blocked · 1 skipped · 12 remaining” and the Pass rate.
- Tabs: Results (the tests), Failures (this run’s failures) and Progress (who did what, and the burndown).
- Live in the header: the page updates by itself when anyone else records a result.
Recording a result
- Click a test. The panel on the right shows its preconditions and steps.
- For each step, choose Passed, Failed, Blocked or Skipped. If a step fails, the overall result switches to Failed for you.
- Choose the overall Result: Passed, Failed, Blocked, Skipped or Rerun.
- Add a Comment (what you saw), optionally the actual result per step, the time (there is a Start timer button) and screenshots.
- Click Save result & next. TestPlum moves to the next test.
Keyboard (much faster)
| Key | Does |
|---|---|
| 1 to 5 | Choose Passed, Failed, Blocked, Skipped, Rerun |
| ⌘ + Enter / Ctrl + Enter | Save and go to the next test |
| J / K | Next / previous test |
| Focus mode button | Hides everything but the test in front of you |
When to use which result
| Result | Use it when | Counts in pass rate? |
|---|---|---|
| Passed | Everything worked as expected | Yes (as a pass) |
| Failed | Something did not work | Yes (as a failure) |
| Blocked | You could not test, for example the environment was down or another bug was in the way | Yes (as not passed) |
| Skipped | You decided not to run it on purpose, for example it does not apply to this build | No, but it counts as done for completion |
| Rerun | It needs to be run again | No, it is not final |
| Not run | Not tested yet | No, and it keeps completion below 100% |
Working as a team
- Filter the list by status or by person (Everyone, or a name).
- Select several tests to assign them, or record the same result for all of them.
- The Inbox (sidebar) shows every test assigned to you in active runs, across all projects, and your open bugs.
Finishing a run
- Rerun failures creates a new run with only the failed and blocked tests, assigned to the same people.
- Sign off completes the run and locks its results, so nobody can change them by accident. Reopen if you must.
- Export CSV and the printer button give you the run’s results to share.
Results are append-only: a new result never erases the old one; the history is kept.
Step 11: Report and track bugs
From a failed test
In the run, open the failed test and click Create bug. TestPlum fills in:
- The title (“Pay with a saved card — fails”), the priority and the run.
- The steps to reproduce, marking the step that failed (
← FAILED) with its actual result. - What you observed (your comment).
Check it, pick the severity and assignee, and click Create bug. The bug (for example NWC-BUG7) is linked
to the test and the result. TestPlum warns you about likely duplicates before you file.
The bug’s life
New ──► In progress ──► Resolved ──► Closed
│ ▲ │
└► Blocked ┘ └► (reopened back to New)
Use Start progress, Mark blocked, Resolve (you choose a resolution, such as Fixed) and Close. Every change is in the bug’s History.
Open critical bugs block the answer. A bug with Critical priority or Blocker/Critical severity keeps the release Not ready until it is resolved.
When the test passes again. Once the tests linked to an open bug pass again, the bug is marked Test passes now: in the bug list (tick Tests pass now to see only those), on the bug page with a Close as fixed button, and under Needs you on the project home. Recording a pass on a test that still has an open bug also reminds you.
Jira and GitHub (two-way)
An administrator connects them under Integrations and API → Issue trackers. Then:
- Send a bug to Jira or GitHub from the bug page.
- When the issue is closed or done there, the bug is resolved here. Reopen it there, and it reopens here.
- Resolve or close it here, and it closes there within seconds.
- The bug page shows the last sync, and Sync now.
Step 12: Explore without a script
Not every problem is found by scripted tests. Use Runs → Sessions for exploratory testing.
- Click New session and fill in:
- Charter: what you are exploring and why (“Explore checkout with slow networks to find payment timeouts”).
- Area, Environment, Timebox (for example 60 minutes) and Tester.
- Optionally the open release and the requirements it explores.
- Start the session. A clock shows the time against the timebox.
- While exploring, add notes, bugs, questions and ideas to the timeline. A bug entry can be filed as a real bug in one click. Attach screenshots.
- Complete the session with an outcome (no problems, problems found, blocked) and a summary. The timeline is then locked.
Sessions show up on their release and their requirements.
Step 13: Send automated test results from CI
Automated results land in the same place as manual results and count toward the same answer.
Connect CI in three steps
Go to Runs → Connect CI:
- Which tool runs your tests? Playwright, Jest, Vitest, pytest, or anything that writes JUnit XML. The page shows how to make your tool write a report file.
- Where do your tests run? GitHub Actions, GitLab CI, or another CI or your computer.
- Make a key and add one step.
- Click Create CI key. Copy it now, it is shown only once. Save it as a secret named
TESTPLUM_CI_KEYin your CI. - Choose what the build should do with the answer: Show the answer in the build log (the build passes or fails on your tests, as now), or Stop the build when Not ready (the build fails until the answer is Ready).
- Copy the step into your pipeline. It runs even when tests fail, so failures are reported too.
- Click Create CI key. Copy it now, it is shown only once. Save it as a secret named
Run your pipeline. The page shows “Results received from CI #418” without a reload, with the counts and the answer.
What the CI key can do: only send results to this project and read its answer. Anything else is refused, so a leaked key exposes very little. Revoke keys on the same page.
What happens with the results
- Each upload becomes an automated run (CI #418). Tests are created automatically the first time they appear and matched by name afterwards.
- Results go to the release you are working on (the open release due soonest).
- Supported formats: JUnit XML, CTRF JSON and Allure results. Retries are kept as separate attempts, so flaky tests show up.
- Each test counts once, by its newest result. An old failing build stops counting as soon as a newer build has the same test, so you never have to clean up old CI runs.
- Each environment counts. A run’s Environment (for example Chrome, Safari or Android) keeps its results apart: a test that fails on Safari is still failing after it passes on Chrome. A retest in the same environment replaces the failure. With more than one environment, the answer shows each one (Safari: 2 failing).
What your build log shows
Not ready: 2 things to fix
For release 5.2 (NWC-REL3).
Not ready since 2026-05-21 14:12 UTC.
1. 1 critical bug is still open.
2. Only 92.3% of tests pass. Needs 95%.
Details: https://app.testplum.com/projects/…
On GitHub the same answer is added to the job summary.
More for developers (Playwright runners, schedules, the Automate button, tests as code, the API and AI assistants) is in For developers.
Step 14: Understand failures and flaky tests
Failures, sorted by cause
Open Failures. Failures from runs in progress and runs signed off in the last two weeks are listed per run, and:
- Grouped by cause. The same problem in ten tests is one group, so you deal with it once.
- Sorted into four kinds, with the reasons and a confidence:
| Kind | Means | Usually fixed by |
|---|---|---|
| Product bug | The app does not do what the test expects | Developers |
| Broken test | The test is out of date or wrong | Whoever owns the test |
| Environment | Servers, network or browsers were not ready | DevOps |
| Flaky test | Fails some of the time with no change | Quarantine, then fix |
If the sorting is wrong, click Change and pick the right kind. TestPlum remembers your choice for the same error next time.
One bug per cause: File bug on a group creates one bug with the error, the reasons and every affected test. The next run with the same failure links to the open bug instead of asking again.
Flaky tests and quarantine
Failures → Flaky tests lists tests that switch between pass and fail, with a flakiness score and their recent results. Tests that fail every time are not flaky; they are broken, and are not listed.
To stop a flaky test from blocking the release while someone fixes it, click Quarantine:
- Why: what makes it flaky (“Lidar timeout on the shared CI runner”).
- Owner: who fixes it (you, if left empty).
- Expires after: 1 to 90 days (default 14).
- Passes in a row that confirm a fix: default 10.
While quarantined, the test counts neither for nor against the answer. The quarantine ends by itself when the test passes enough times in a row, or when it expires (the owner is told). A quarantined critical test still keeps the release Not ready, so you cannot hide a critical problem.
Fixes with proof (optional)
For a broken automated test, TestPlum can prepare a fix that is proven to pass before anyone sees it, and that is never allowed to simply hide the failure. A person reviews it under Failures → Fixes and approves or rejects it; TestPlum never changes your code itself. This needs a runner and AI switched on.
Step 15: Get the answer: can we ship?
Open the project. The home shows:
● CAN WE SHIP?
Not ready: 3 things to fix
5.2 · NWC-REL3 · build 5.2.0-rc1 · due 30 May
5 of 8 checks pass · not ready since 4 hours ago · checked now
⏱ Behind: 40 tests left take about 4 days at this week's pace (10 a day), but 5.2 is due in 2 days.
What to fix to turn it green
1 1 critical bug is still open. [Open bugs →]
2 Only 86.7% of tests pass. Needs 95%. [Sort the failures →]
3 Only 76.9% of tests have a result. Needs 100%. [Finish the open runs →]
▸ What changed: Turned Not ready 4 hours ago
🔔 Your team hears when it turns, in 1 channel. [Tell me when it turns]
- One sentence per problem, most blocking first, each with a button to where you fix it.
- What changed lists each turn: when the light turned, which reasons were New and which were Fixed, and the newest results at the time.
- If more than one release is open, a picker lets you look at another release.
- Will we finish in time? For a release with a due date, a line says whether the tests still without a result finish before it at the current pace: On track, Behind, Stalled (no results this week) or Overdue. It is a forecast to plan with, not one of the checks, so it never changes the light. The build-log answer and assistants get the same line.
- Since the last release. For a release, the answer says which tests passed in the release you shipped before but fail now, and how many were fixed since, each opening in its run. The build log, the morning summary and assistants say it too.
- Run these first. While tests are still left to run, the answer names a few of them that are most likely to change it, each with the reason (for example, it failed last time or has an open bug). Each opens that test in its run, so the surprises show up early instead of the day before the release. Assistants get the same list.
- Who has what left. Beside it, each person’s tests still to run, and how many nobody has yet. Leads click Share out, pick the people, and the tests nobody has are split evenly, counting what each person already has. Everyone is told what they got.
- Under the answer, four steps show the numbers behind it: Tests, Results, Analysis (failing tests, flaky tests, open bugs) and Verdict. Needs you lists what is waiting, most urgent first: failing tests to sort, overdue runs, open critical bugs and tests still to run.
How the answer is worked out
- It looks at the runs of the release (or, with no open release, all open runs).
- Each test counts once per environment, by its newest result there. If a test failed on Monday and passed a retest on Wednesday in the same environment, it counts as passed; a pass on Chrome does not cancel a failure on Safari.
- Skipped tests are left out of the pass rate.
- Each check is compared with its target (see Question 5).
The trusted pass rate: not every pass proves the same
A green test suite is not always proof. A test that checks nothing, or passes only on a retry, proves less than it looks. So TestPlum also works out a trusted pass rate, where strong passes count in full and weak ones count for less. Passes count for less when, for example:
- the automated test has no assertions (it checks nothing);
- it passed only on a retry, or the test is flaky;
- the test changed after it ran;
- it was written by AI and nobody has reviewed it yet;
- it is a manual pass without an expected result.
When the trusted pass rate is below its target, the release page lists the weakest passes and how to make each one count fully (for example, add an expected result, or review an AI-written test).
The release page
Releases → 5.2 shows every check with its evidence, target and a link to resolve it, the runs in the release, the requirements and the weakest passes. Print report gives a page for the go/no-go meeting with sign-off lines.
Step 16: Ship the release
When the light is green:
- Open the release and click Ship. Add a note (“Shipped to production after the payment fix”). The verdict at that moment is recorded with your name and the time.
- If you must ship while Not ready, TestPlum asks for a reason. The override is recorded in the audit log.
- A shipped release is read-only and takes no new runs. Releases can also be cancelled and reopened.
After shipping, the home answers for the next open release. Create it (5.3) if you have not yet.
Electronic signatures and the validation pack (Business plan)
For regulated teams:
- A release can require up to five signatures, each with a meaning (“Reviewed by QA”, “Approved for release”), each by a different person who enters their password again (or signs in with SSO again).
- A signature is tied to the release exactly as it was. If anything changes afterwards, the signature stays on file but no longer counts, and must be given again.
- Download the validation pack: one zip with the verdict, signatures, traceability, every result with its evidence, the audit trail and checksums, ready for an auditor.
Step 17: Keep everyone informed
You do not need everyone to open TestPlum to know where things stand.
| How | Who it is for | Set up |
|---|---|---|
| Tell me when it turns | Anyone who wants to know | Click the button under the answer. You get a notification and an email each time the light turns. Stop telling me to undo. |
| Slack or Microsoft Teams | The whole team | Project settings → Notifications → Slack and Microsoft Teams. Paste the channel’s incoming-webhook URL, pick events: a run is signed off, automated results include failures, the answer turns Ready or Not ready, and, if you want it, a summary every weekday morning (at 9:00 in your time zone: the answer, whether the release finishes in time and which tests to run first). Click Send test. |
| Status badge | READMEs and wikis | Project settings → Notifications → Status badge → Turn on the badge, then copy the Markdown or HTML. It shows only TestPlum: Ready or Not ready, through a private link you can renew or turn off. |
| Share the answer | People without an account (a product owner, a customer, a manager) | Project settings → Notifications → Share the answer → Turn on sharing, then copy the link. The page shows the light, the release and its due date, the reasons in plain words, whether the tests left finish in time and each environment, and updates on its own. It never shows test names, people, bugs or results. Anyone with the link can open it, so treat it like a password: New link replaces it, Turn off stops it. |
| Projects page and Portfolio | Managers and executives | Each project card and the Portfolio table show each project’s answer and first reason, and “1 of 4 projects are ready to ship”. |
| Webhooks | Your own tools | Integrations and API → Webhooks. Events such as the answer turning, a run being completed, a result being recorded or a bug being created. Signed so you can verify them. |
| Notifications (bell) | Everyone | Mentions, assignments, follows and finished background tasks. |
Each turn of the light is announced once, including when it turns because a release was shipped and the home moved on to the next one.
Step 18: Reports and insights
Insights (Releases → Insights)
| Tab | Shows | Use it to |
|---|---|---|
| What to test next | Tests ranked by risk, each with the reasons. | Click Create run from these tests for a smart, short run when time is tight. |
| Stability | Tests whose results flip, with a flaky score. | Find tests to quarantine or fix. |
| Workload | Per person: open assignments, planned effort, results and time logged in 30 days. | Balance work in a sprint. |
| Custom | Build your own chart or table: tests, results, run assignments or bugs, grouped by anything. Save it for everyone. | Bugs by status per squad, Results per day. |
Other reports
- Run → Progress: who did what, and the burndown against the deadline.
- Run → Export CSV / Print: share results.
- Release → Print report: the go/no-go document.
- Portfolio (sidebar): every project’s answer, plus custom reports across projects.
- Activity: everything that happened in the project, by whom.
Step 19: Keep the test suite healthy
A big product collects old and duplicate tests. Tests → Maintenance helps:
- Likely duplicates: pairs of tests that look the same, with the reasons and a suggestion of which to keep. Merge moves requirement and bug links, tags and the automation ID to the kept test and sends the other to the deleted items. Not duplicates is remembered.
- Stale tests: tests that have not run for 30 to 365 days (you choose), are always skipped, never ran, or whose automation went quiet. Keep them, or move them to the deleted items in bulk.
Deleted items (Organization settings → Governance) hold deleted tests, runs and other items so you can bring them back.
Good habits for a big suite
- Review Stale tests every month and Likely duplicates every quarter.
- Sign off runs when they are done.
- Keep Critical for tests that would really stop a release.
- Link every new story to tests; fix Not covered requirements at sprint planning.
For developers: automation, API and AI assistants
Results from any CI
See Step 13. One pasted step and a CI key is enough; nothing to install. Results go to the release you are working on, or to a release you name.
Automate a manual test
On a manual test, click Automate. TestPlum writes a Playwright test file with one test.step per step and the
test’s automation ID, so its results come back to the same test. Git export turns several tests into files in
your repository at once, as a patch or a new branch.
Playwright on your own runners
Under Runs → Automation, an administrator adds a runner (a small program you start on your own machine or CI box). Point it at a repository and run its Playwright tests on demand, on a schedule (for example nightly) or for every pull request. Screenshots, videos and traces come back with the results, and the log streams live.
Run only the tests that matter
Test selection suggests which automated tests to run for a code change, so pull requests get feedback faster. The Automation page shows how well it would have worked on your past builds.
Tests as code
Tests can live as Markdown files in your repository next to the code, synced both ways with TestPlum: front matter for fields, sections for preconditions and steps. A test changed on both sides is reported as a conflict and never overwritten, and a check can fail a pull request on it.
The TestPlum command-line tool (for test selection, tests as code and the CI gate), the Playwright reporter and the MCP server for AI assistants are available on request. Contact the TestPlum team. Everything they do is also available through the REST API.
CI gate and commit statuses
A pipeline can ask whether a release (or a single run) is ready and stop a merge or deploy if not. The answer is recorded with the commit, and the release page shows which commits were held back and why. The simple way is the Stop the build when Not ready option on Connect CI.
REST API and webhooks
- API keys: Integrations and API → API keys. A key is shown once; revoke it any time.
- Everything in the app can be done through the documented REST API, including asking for the answer (also as plain text for a build log). See the developers page.
- Webhooks are signed so you can check they came from TestPlum, are retried if your system is down, and their deliveries can be inspected and retried.
AI coding assistants (MCP)
TestPlum’s MCP server connects assistants such as Claude Code, Cursor and VS Code. The assistant can read
requirements, write tests linked to them, record results, sort a run’s failures and answer “can we ship?” with the
can_we_ship tool, giving the same answer and reasons as the home, for one project or every project. AI-written tests
count only partly until a person reviews them. Nothing an assistant does can delete data.
For administrators: security, plans and housekeeping
Organization settings
| Group | Tabs |
|---|---|
| People and access | Members, Roles, Single sign-on |
| Workflow | Fields, Field layouts, Result names, Test types |
| Governance | Audit log, Deleted items |
| Plan | Your plan, use against limits, trial |
Single sign-on (Business plan)
Connect any OpenID Connect provider (Okta, Microsoft Entra ID, Google Workspace, Auth0…) and prove you own your email domain. People on that domain then see Continue with SSO. You can choose to add new people automatically or by invitation only, and you can require SSO for the domain.
Audit log
Important actions are recorded, such as sign-ins, role changes, keys, shipping overrides, signatures and deletions. Business plan administrators can download it as CSV.
Your own account
Account: name, time zone, Change password, Active sessions (sign out other devices) and Delete account.
Plans and limits
| Plan | For | Limits and extras |
|---|---|---|
| Free | A small team getting started | Up to 3 testers, unlimited viewers, 1 project, unlimited automation results; releases, CI gate, flaky quarantine and AI (fair use) included |
| Team | Teams that ship every week | Everything in Free, unlimited testers and projects |
| Business | Organizations with a security review | Everything in Team, single sign-on, audit log export, electronic signatures, priority support |
| Enterprise | Regulated and self-hosted installs | Everything in Business, self-hosting |
- Testers are owners, admins and anyone whose project role can change tests, runs or bugs. Viewers are free.
- An owner can start one 14-day Business trial under Plan.
- Going over a limit never deletes anything; adding more is refused with the reason until you upgrade or remove some.
A whole sprint, day by day
Here is Sprint 24 at Northwind, for release 5.2, so you can see how everything fits together.
Day 1 (Monday): sprint planning
- The QA lead opens Releases and checks that 5.2 is open with the right target date and thresholds.
- Tests → Requirements → Import from Jira with
sprint = "Sprint 24". Six new stories appear as Not covered. - The squad writes or links tests for each story. Two stories get new tests (“Pay with Apple Pay”, “Save a
second card”), tagged
sprint-24andcheckout-squad. - Runs → New run: Sprint 24 – Checkout, release 5.2, deadline Friday next week, Suggested tests plus
the new ones (By filter: tag
sprint-24). Tests are assigned to the three testers.
Days 2–8: testing
- Testers open the Inbox, work through their tests with the keyboard (1, ⌘+Enter), and file bugs from failures. NWC-BUG21: Apple Pay sheet does not open on Safari is Critical.
- Every pull request runs Playwright in GitHub Actions; results arrive as CI #4xx runs in release 5.2.
- The team’s Slack channel gets “Northwind Commerce: Not ready to ship” when the critical bug turns the light red.
- On Failures, a group of 6 failing API tests is sorted as Environment (“connection refused”); DevOps fixes the staging database. A Playwright test that flips is quarantined for 14 days with an owner.
- The QA lead checks Run → Progress: the burndown is behind, so two tests are re-assigned.
Day 9: bug fixes and retest
- Developers fix NWC-BUG21 and resolve it (in Jira; it resolves here too).
- The tester opens the run, filters to Failed, and records the retests. Each retested test now counts by its newest result, a pass.
- Rerun failures is used for anything still failing, to keep a clean list.
Day 10: wrap up
- The run is signed off. The home shows Ready to ship for 5.2, and “Turned Ready” in What changed. Slack says “Northwind Commerce: Ready to ship”.
- Before the release, the Release full regression cycle is started: Start run from cycle. When it is done and the light is still green, the release manager prints the report, gets the signatures, and clicks Ship.
- The QA lead creates release 5.3, so the home starts answering for the next one.
Checklists for each role
Tester, every day
- Open the Inbox and work through your assigned tests.
- Record each result with a comment when it is not a plain pass.
- File a bug from every real failure, and check the suggested duplicates first.
- Mark Blocked (not Failed) when you cannot test.
QA lead, every week
- Check the project home: is the light right, and are the reasons expected?
- Sort new groups on Failures; quarantine flaky tests with an owner.
- Look at Run → Progress and re-balance work.
- Make sure every story in the sprint has tests (Requirements, filter Not covered).
- Sign off finished runs.
Release manager, before shipping
- The release’s light is green, with no overrides needed.
- No quarantined critical tests and no open critical bugs.
- Print report for the go/no-go meeting; collect signatures if required.
- Ship, then open the next release.
Developer
- Add the Connect CI step to your pipeline once.
- Read the answer in the build log on your pull requests.
- Fix the bugs assigned to you; resolve them in Jira or TestPlum.
- Use Automate to turn stable manual tests into Playwright tests.
Product manager or executive
- Check the Projects page or Portfolio: which projects can ship?
- Click Tell me when it turns on the projects you care about.
Common questions and problems
The answer says “Add your first tests” or “Record your first results”. There is nothing to judge yet. Follow the steps shown: add tests, then record results. There is no red light until there are results.
The answer is about the wrong release (or about “all open runs”). The home answers for the open release due soonest. Check the target dates under Releases, use the picker on the home to look at another release, or create a release if none is open.
A test failed, then passed on a retest, but the answer still looks bad. Each test counts by its newest result in the release. Make sure the retest run belongs to the same release (pick the release when you create the run, or Choose runs on the release page).
“Only 76.9% of tests have a result. Needs 100%.” Some tests in the release’s runs are still Not run. Record them, mark them Skipped if they do not apply, or remove them from the run’s selection.
The trusted pass rate is low but the pass rate is high. Some passes are weak, for example tests that check nothing or pass only on a retry. The release page lists the weakest passes and what to do about each.
I cannot create a second project / invite another tester. The Free plan allows 1 project and 3 testers. Viewers are free. Start the 14-day Business trial or upgrade under Organization settings → Plan.
Invitations or password resets do not arrive. Check the spam folder first. For an invitation, an admin can Copy new link under Pending invitations and send it another way. For a password reset, request a new link from the sign-in page, or contact TestPlum support.
A run’s results cannot be changed. It was signed off (locked). Click Reopen if you really need to change it.
I deleted something by mistake. Look in Organization settings → Governance → Deleted items.
Importing the same CSV again created copies. Add an ID column with your own IDs. Tests with an ID that was imported before are left as they are.
Cheat sheet
Keys
| Key | Where | Does |
|---|---|---|
| ⌘K / Ctrl+K | Anywhere | Search or jump to anything |
| 1–5 | Running a test | Passed, Failed, Blocked, Skipped, Rerun |
| ⌘/Ctrl + Enter | Running a test | Save the result and go to the next test |
| J / K | Running a test | Next / previous test |
| ⌘/Ctrl + Enter | Writing steps | Add a step |
Keys of items
| Item | Looks like |
|---|---|
| Test | NWC-C42 |
| Requirement | NWC-R12 |
| Run | NWC-RUN31 |
| Bug | NWC-BUG7 |
| Release | NWC-REL3 |
Where things are
| I want to… | Go to |
|---|---|
| See if we can ship | Project Home |
| Write or import tests | Tests → New test / Import |
| Link stories to tests | Tests → Requirements |
| Start testing | Runs → New run, or Run tests on Home |
| Repeat a regression | Runs → Cycles |
| Explore freely | Runs → Sessions |
| Send CI results | Runs → Connect CI |
| Sort failures | Failures |
| Handle flaky tests | Failures → Flaky tests |
| Track bugs | Failures → Bugs |
| Plan and ship a release | Releases |
| See trends and risks | Releases → Insights |
| Clean up old tests | Tests → Maintenance |
| Alerts and README badge | Project settings → Notifications |
| My work across projects | Inbox |
| All projects’ answers | Portfolio |
| People, roles, SSO | Organization settings → People and access |
| Fields, result names, types | Organization settings → Workflow |
| API keys, webhooks, Jira/GitHub, runners | Integrations and API |
This guide describes TestPlum as of November 2026. Where a feature depends on your plan (single sign-on, signatures, audit log export), the guide says so.
Frequently asked questions
Where do I start with TestPlum?
Create a free workspace. To look around first, click Try with sample data for a filled-in sample project. Then create a project for your product and follow the three steps on its home: add your tests, record results, get your answer.
How should a large product with many teams be set up?
Use one project per product that ships on its own schedule, folders by feature area, tags for things that cut across folders such as smoke or a squad, releases for each version you ship, and runs for each sprint or regression.
Do I need to install anything?
No. TestPlum runs in the browser. Automated results come from your CI with one pasted step and a key that only works for one project.
Can people who only look at results use TestPlum for free?
Yes. Viewers are free on every plan. Only testers, people who can change tests, runs or bugs, count toward the plan.