03 Failures
Understand every failure, not just count them
TestPlum groups failures that share a cause, so ten tests failing for one reason are one problem. Each group is sorted as a product bug, a broken test, an environment problem or a flaky test, with the reasons shown. You file one bug per cause, and for a broken automated test TestPlum can propose a fix and prove it passes before a person approves it.
What's included
Failures grouped by cause and sorted with reasons, one bug per cause, and fixes proven before approval.
- 03.1
Grouped by cause
Failures with the same underlying error become one group, across tests and runs. Numbers, ids and stack traces never split a group.
- 03.2
Sorted, with the reasons
Each group is a product bug, a broken test, an environment problem or a flaky test, decided by visible rules with a confidence. Correct it once and the same failure arrives sorted next time.
- 03.3
One bug per cause
File a bug for the whole group with the error, the reasons and every affected test, then send it to GitHub or Jira. The next run with the same failure links to the open bug.
- 03.4
Fixes with proof
For a broken automated test, a runner reproduces the failure, AI proposes a change, and guardrails refuse anything that would hide a failure. The fix must pass every time, with before and after screenshots.
- 03.5
A person decides
Only a signed-in person approves a fix, and a changed expected value needs explicit sign-off. You apply the patch; TestPlum never writes to your repository.
- 03.6
Flaky tests in the open
Tests that switch between pass and fail are scored, shown on their own and counted only partly in the release verdict.
In practice
How it goes
Open Failures
See every failure from runs in progress and runs signed off in the last two weeks.
Confirm the cause
Each group arrives sorted with its reasons; confirm it or correct it with a note.
Act on it
File one bug for a product bug, fix a broken test, rerun an environment problem or mark a flaky one.
Approve with proof
Review a proposed fix's diff, guardrails and screenshots, and approve or reject it.
How does TestPlum decide whether a failure is a product bug or a broken test?
By visible rules: environment errors such as timeouts and refused connections first, then tests that passed on retry or have a flaky history, then errors in the test code itself, then failed checks, which point at the product. Every decision shows its reasons and a confidence, and a person's correction is remembered.
Can AI change my tests?
It can only propose. A runner reproduces the failure, guardrails refuse a change that drops assertions, skips tests or removes the test, a changed expected value is flagged, and the fix must pass every time. A person approves it and applies the patch; TestPlum never writes to your repository.
Do I need AI to sort failures?
No. Grouping and sorting are rule-based and work without AI. AI adds an optional second opinion and proposes fixes when an administrator switches it on.
The rest of the loop
Find out whether your next release is ready
Create a workspace, import your tests or write a few, and record a run. Everything on this site is in the product today.