How to quarantine flaky tests without hiding real bugs
How to quarantine flaky tests: detect them from run history, set an owner and expiry, keep them running in a non-blocking CI job and define exit criteria.
Quarantine a flaky test by taking its results out of what blocks merges and releases while it keeps running and recording results. Only quarantine tests that flip between pass and fail, never ones that fail every time, and give each quarantine a reason, an owner, an expiry and an exit rule such as ten passes in a row.
What quarantine is, and what it is not
Quarantine means taking a flaky test out of the set of results that can block a merge or a release, while it keeps running and keeps recording results. It is a holding area with a named owner and a deadline, not a place where tests go to be forgotten.
It is not the same as:
- Deleting the test. The coverage it gave is gone, usually for good.
- Skipping the test. A skipped test produces no results, so you can never prove it has been fixed.
- Retrying until green. Retries hide the flakiness instead of isolating it, and they also hide real intermittent bugs.
This guide is the process. If you first want to understand why tests flake and how to score flakiness, read flaky tests: how to detect, measure and fix them.
What counts as flaky
A test is flaky when it passes and fails on the same code without a relevant change. Two signals in your run history point to it:
- Pass and fail flips. Across recent runs, the outcome keeps changing between pass and fail.
- Pass on retry. The test failed, was retried in the same build and then passed. Nothing changed between the two attempts, so one of them was wrong.
Just as important is what does not count as flaky. A test that passed for weeks and then fails every time is broken: something changed, in the product or in the test. That is a regression to fix, and it must never be quarantined, because quarantine would hide exactly the failure the test exists to catch.
| Recent results (oldest → newest) | What it is | Quarantine? |
|---|---|---|
| P P F P P F P F P P | Flaky: flips back and forth | Yes, if it blocks people |
| P P P P P P P F F F | Broken: started failing and stays failing | No, fix it now |
| P P F (passed on retry) P P | Flaky: one pass on retry | Watch it; quarantine if it repeats |
| F F F F F F | Broken, or a test that never worked | No |
Detect flakiness from run history
You do not need guesswork. Use the flip score from the flaky tests guide on each test’s recent decisive results (pass or fail; ignore skipped and blocked):
Flaky score = flips ÷ (decisive results − 1)
Only score a test that has at least three decisive results including both a pass and a fail. Then apply two rules:
- Suggest for quarantine when the score is at or above about 0.3 over the last ten runs.
- Refuse quarantine when the last three decisive results are all failures. That test is broken, not flaky.
For the pass-on-retry signal, your results have to record every attempt, not just the final one. If your CI only keeps the last attempt, a test that needed two tries looks identical to one that passed first time. The JUnit XML format, CTRF and Allure can all carry results to a test management tool; check that your reporter keeps retries as separate entries.
TestPlum uses exactly these rules: its quarantine suggestions list tests with a score of 0.3 or more over the last 10 runs, and it refuses to quarantine a test whose last three decisive results are all failures. See failure triage for how flaky failures are sorted from product bugs, broken tests and environment problems.
The quarantine rules
Write these down before you quarantine anything, and agree them with the people who will feel the effect: developers, QA and whoever owns the release.
| Rule | Why | A sensible default |
|---|---|---|
| Every quarantine has a reason | So the next person knows what was seen | One sentence plus a link to the failing runs |
| Every quarantine has an owner | Unowned quarantines never end | One named person, not a team |
| Every quarantine has an expiry | Forces a decision instead of drift | 14 days, at most 90 |
| The test keeps running | No results means no proof of a fix | Every build, in a non-blocking job |
| Results are recorded | History is how you exit | Same tool, same test ID as before |
| Critical tests stay visible | A flaky test on a critical path is still a risk | The release is not ready while one is quarantined |
| Exit is automatic | People forget to lift quarantines | A streak of passes ends it |
Two of these deserve a little more explanation.
The expiry. When a quarantine expires, the owner has to choose: extend it with a reason, fix the test, or delete it on purpose. An expiry turns a silent backlog into a list of decisions.
Critical tests. Quarantine removes a test’s results from the pass rate, but it should not remove the risk from the release decision. If the checkout test is quarantined, the honest status of checkout is “not verified”. Treat a quarantined critical test as a reason the build is not ready yet.
Keep quarantined tests running in CI
The usual pattern is two jobs: one that runs the healthy tests and blocks, and one that runs the quarantined tests and does not.
GitHub Actions supports this directly. Setting continue-on-error: true on a job “prevents a workflow run from failing when a job fails”, according to the workflow syntax reference. There is also a step-level continue-on-error, which lets a job pass when one step fails (reference).
To split the tests, mark quarantined ones with a tag. In Playwright, a test can carry a tag such as @quarantine, and the CLI can run only tagged tests with --grep or everything else with --grep-invert (Playwright tag docs). Playwright writes JUnit XML when you pass --reporter=junit and set PLAYWRIGHT_JUNIT_OUTPUT_NAME (Playwright reporters).
name: tests
on: [push, pull_request]
jobs:
tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
- run: npm ci
- run: npx playwright install --with-deps
- run: npx playwright test --grep-invert @quarantine
quarantined:
runs-on: ubuntu-latest
continue-on-error: true # a failure here never fails the workflow run
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
- run: npm ci
- run: npx playwright install --with-deps
- run: npx playwright test --grep @quarantine --pass-with-no-tests --reporter=junit
env:
PLAYWRIGHT_JUNIT_OUTPUT_NAME: quarantine.xml
- name: Send results to TestPlum
if: always() # send results even when the tests failed
run: |
curl -f -sS -X POST \
-H "Authorization: Bearer $TESTPLUM_CI_KEY" \
-H "Content-Type: application/xml" \
--data-binary @quarantine.xml -o /dev/null \
"https://app.testplum.com/api/v1/projects/<project-id>/automation/report?release=%40next&run_name=Quarantine%20%23${{ github.run_number }}"
env:
TESTPLUM_CI_KEY: ${{ secrets.TESTPLUM_CI_KEY }}
A few notes on this example:
--pass-with-no-testsstops the job failing on an empty quarantine. Playwright documents it as making the run “succeed even if no tests were found” (CLI options).if: always()on the upload step makes it run after a failed step. Thealways()status function is described in GitHub’s expressions reference.- Upload the healthy job’s results the same way, so both land in one history. See GitHub Actions with TestPlum for the full setup.
Tags in code are one way to mark tests. The other is to keep the list outside the code: TestPlum’s quarantine list includes each test’s automation ID, so a CI script can read it and exclude those tests without a code change. Pick one source of truth; two lists will drift apart.
Exit criteria: how a test leaves quarantine
A quarantine should end in one of three ways, and all three should be visible.
- Fixed and confirmed. Someone fixed the root cause, and the test then passed a number of times in a row. Ten consecutive passes is a reasonable default; raise it for tests that used to fail rarely. A single failure resets the count.
- Expired. The deadline passed without a confirmed fix. The owner is told and decides to extend, fix or delete.
- Lifted by hand. Someone lifts it, for example because the test was rewritten. Record who and why.
Do not end a quarantine on “I think I fixed it”. A test that flaked one run in five can easily pass five runs in a row by luck, which is why the streak needs to be long enough to be convincing.
In TestPlum a quarantine has a reason, an owner, an expiry of 1 to 90 days (14 by default) and a number of passes in a row that confirm the fix (10 by default). It ends by itself on a confirmed fix or on expiry, and the owner is notified either way.
How to stop quarantine hiding real bugs
Quarantine is safe only if it is hard to abuse. These guardrails do most of the work:
- Never quarantine a broken test. Check the last few decisive results first. If they are all failures, it is a regression.
- Look at the failure before you quarantine. If the error is a failed assertion on product behaviour, such as a wrong total or a missing element, it may be a real intermittent bug, like a race condition in the product. Those are flaky in the test and in production. File a bug, do not just quarantine.
- Keep the error messages. Group failures by their error. If a quarantined test starts failing with a new error, that is new information and deserves a look.
- Show the gap in coverage. A quarantined test should count as “not verified” for its requirement, not as passing and not as invisible. The requirements traceability matrix guide explains how requirement status is derived.
- Cap the list. If quarantine keeps growing, the team is quarantining instead of fixing. Agree a limit and treat hitting it as a signal to stop and spend time on test health.
- Review it weekly. Five minutes in a standing meeting: new entries, expiring entries, entries with no recent results.
Metrics for your quarantine
Measure the quarantine itself, not just the tests in it.
| Metric | How to calculate | What it tells you |
|---|---|---|
| Quarantine size | Tests in quarantine right now | Whether flakiness is under control |
| Share of suite quarantined | Quarantined ÷ active automated tests | Size in context; a rising share is a warning |
| Time in quarantine | Days from start to end, per test | Whether owners actually fix things |
| Exit mix | Fixed vs expired vs lifted, last 30 days | Expired-heavy means quarantine is a parking lot |
| Re-entry rate | Tests quarantined again within 30 days of leaving | Fixes that did not hold |
| Flaky score after exit | Score over the next 10 runs | Proof the fix worked |
There is no industry benchmark worth quoting for these. Track your own trend and agree what “too many” means for your team.
Quarantine and the release decision
Quarantined tests should drop out of the pass rate, because a coin-flip result tells you nothing either way. But the release decision still has to account for them:
- They lower execution completion: the cases exist but have no trustworthy result.
- A quarantined critical test should keep the verdict at not ready, or need an explicit, recorded override.
- Passes from flaky tests that are not quarantined should count for less, because they prove less. That is the idea behind a trusted pass rate, explained in QA metrics.
TestPlum’s verdict follows these rules: quarantined tests are left out of every count, show as untested for coverage, and a quarantined critical test keeps the release at not ready. See release readiness and the go/no-go checklist template.
Frequently asked questions
What is flaky test quarantine?
It is moving a flaky test out of the results that can block a merge or a release, while it keeps running in a non-blocking job and keeps recording results. Unlike skipping or deleting, the test still produces evidence, so you can prove when it has been fixed.
Should you quarantine a test that fails every time?
No. A test that fails on every recent run is broken, not flaky: something changed in the product or the test. Quarantining it would hide exactly the failure it exists to catch, so fix it or file a bug instead.
How long should a test stay in quarantine?
Set an expiry when you quarantine it; two weeks is a common default, with a hard upper limit such as 90 days. When it expires, the owner must fix it, extend it with a reason, or delete the test on purpose.
How do you know a quarantined test is fixed?
Require a streak of consecutive passes after the fix, for example ten in a row, where any failure resets the count. A single green run proves little for a test that only failed occasionally.
How do you keep quarantined tests running in GitHub Actions without failing the build?
Run them in a separate job with continue-on-error: true, which GitHub documents as preventing the workflow run from failing when that job fails. Upload their results with a step that uses if: always() so the history keeps growing.
Do quarantined tests count toward release readiness?
Their results should not count toward the pass rate, but they should not vanish either: they lower completion and show as unverified coverage. A quarantined critical test should keep the release at not ready unless someone records an override.