QA metrics: what to measure, how to calculate it and what misleads
Software testing metrics with formulas: pass rate, trusted pass rate, completion, escape rate, defect density, flaky rate and go/no-go decisions.
The QA metrics worth tracking are the ones that change a decision: execution completion, pass rate, trusted pass rate, requirements verified, critical failures and open critical defects for the release decision, plus defect escape rate, flaky rate, time to triage and test duration for improving the process. Agree thresholds before testing, and ship only when every check passes.
What QA metrics are for
QA metrics are numbers about your testing and your product’s quality that help a team make a decision: what to test next, where quality is slipping, and whether a build can ship. A metric earns its place only if someone would act differently when it moves.
That rules out a lot of numbers that are easy to collect, such as “tests written this sprint” or “bugs filed per tester”. Those measure activity, not quality, and they reward the wrong behaviour.
This guide gives definitions and formulas for the metrics most teams actually use, explains which ones mislead, and shows how to turn a handful of them into a go/no-go decision.
Ground rules before you calculate anything
A few definitions keep every formula below honest:
- Decisive result: a final pass or fail (and, for some metrics, blocked). Skipped and untested are not decisive.
- Executed: a test case in scope that has a final status.
- In scope: the test runs that count for this decision, such as the regression run for this release candidate plus the CI suite.
- Latest result: when a test ran several times in scope, use its most recent result, not all of them.
Always show the counts behind a percentage. “47 of 81 executed” is far more useful than “58%”, and it makes a small sample obvious.
The core metrics, with formulas
| Metric | Formula | What it tells you |
|---|---|---|
| Execution progress (completion) | Executed cases ÷ cases in scope | How much of the plan has a result |
| Pass rate | Passed ÷ executed | How much of what ran succeeded |
| Trusted pass rate | Sum of each pass’s trust weight ÷ executed | How much of what ran convincingly succeeded |
| Requirements coverage | Requirements with at least one linked test ÷ all requirements | What is tested at all |
| Requirements verified | Requirements whose linked tests all pass ÷ all requirements | What is proven to work |
| Defect escape rate | Defects found after release ÷ all defects found for that release | How many problems testing missed |
| Defect density | Defects found ÷ size of the component or change | Where defects cluster |
| Flaky rate | Tests with a flaky score ≥ your threshold ÷ automated tests with enough history | How much of the suite you cannot trust |
| Mean time to triage | Average time from a failure being recorded to its cause being decided | How fast failures turn into action |
| Test duration | Wall-clock time of a suite or run (also per test) | How long the feedback loop is |
The rest of this guide goes through each one: how to read it and where it goes wrong.
Progress and pass rate
Execution progress answers “how far along are we?”. Count blocked as executed if you want to see effort, but remember a blocked critical test is not verified. Skipped and untested cases must lower progress; they must never improve the pass rate.
Pass rate answers “of what ran, how much passed?”. It is the most quoted QA metric and the easiest to misread:
- 96% looks healthy until you notice that the 4% is every payment test.
- A pass rate over 30 executed cases out of 400 planned says almost nothing.
- Re-running only the failures after a fix and calculating pass rate over that run alone inflates it. Use each case’s latest result across the whole scope.
So pair pass rate with completion, and with a rule that no critical test may be failing or blocked.
Trusted pass rate: not every pass proves the same thing
A pass is evidence, and some evidence is weak. Consider:
- An automated test that ran no assertions. It passes without checking anything.
- A test that passed only on retry after failing on the first attempt.
- A test with a flaky history, which would have passed or failed regardless.
- A test that changed after it ran, so the pass is for an older version of the steps.
- A manual pass with no written expected result, which is a judgment call.
An ordinary pass rate counts all of these as full passes. A trusted pass rate gives each pass a weight between 0 and 1 based on how much it proves, adds up the weights, and divides by the number of executed cases. Failures count as zero, as they do in pass rate. The result is always at or below the plain pass rate, and the gap between the two is the interesting part: a big gap means a green dashboard is resting on weak evidence.
There is no standard set of weights. Choose yours, write them down, and show the reason next to each discounted pass so nobody has to trust a black box. As a worked example, TestPlum uses these weights, which multiply when several apply:
| Why the pass is weak | Weight |
|---|---|
| Automated test ran no assertions | 0.1 |
| Passed only on retry | 0.5 |
| Flaky test (flaky score ≥ 0.3) | 0.5 |
| Test changed since this run | 0.6 |
| Written by AI and not yet reviewed by a person | 0.6 |
| Manual pass without an expected result | 0.7 |
So 10 executed cases with 9 passes, one of which passed only on retry, give a pass rate of 90% and a trusted pass rate of (8 + 0.5) ÷ 10 = 85%. TestPlum’s release verdict checks the trusted pass rate against its own threshold, 90% by default, so a 100% pass rate made of weak passes can still be “not ready”. See release readiness for how it is shown.
Coverage of requirements
Requirements coverage is the share of requirements with at least one linked test. It tells you what is tested at all, not what works. A requirement covered by one trivial test counts the same as one covered by twenty careful ones.
Requirements verified is the stronger number: the share of requirements whose linked tests have all passed in scope. Use it for release decisions, and add a hard rule that no requirement may be failing.
Two caveats. First, coverage depends entirely on how requirements are written; splitting one requirement into five changes the percentage without changing any testing. Second, code coverage (lines executed by automated tests) is a different metric, useful to developers, and not a substitute for requirement coverage. The requirements traceability matrix guide shows how to build the links behind both requirement numbers.
Defect metrics: escape rate and density
Defect escape rate compares defects found after release, by customers or monitoring, with all defects found for that release:
Escape rate = defects found after release ÷ (defects found before release + defects found after release)
It is the closest thing QA has to an outcome metric, because it measures what testing missed. Its problems are practical: you only know it weeks after release, and it depends on everyone tagging defects with the release they belong to. Agree a fixed window, such as 30 days after release, so releases are comparable.
Defect density divides defects by some measure of size: per component, per thousand lines changed, per feature or per story point. It is useful for one thing, spotting where defects cluster, and it misleads in several ways:
- Size measures are rough. Lines of code reward verbose code; story points are not comparable across teams.
- A low density can mean good code or poor testing. The two look the same.
- It invites comparison between teams, which quickly turns into teams arguing about what counts as a defect.
Use density to compare a component with its own past, not with other teams.
Flaky rate, mean time to triage and test duration
Flaky rate is the share of automated tests that flip between pass and fail without a relevant change. Score each test from its recent decisive results (flips divided by results minus one; see flaky tests) and count those above your threshold. Only include tests with enough history, at least three decisive results, or new tests will distort the number. A rising flaky rate is an early warning that people will soon stop believing red builds. The flaky test quarantine playbook covers what to do about it.
Mean time to triage is the average time between a failure being recorded and someone deciding its cause: product bug, broken test, environment problem or flaky test. It matters because untriaged failures are invisible risk. Measure it in working hours and look at the slowest failures, not just the mean, because a few failures left for a week are what actually hurt. Grouping failures by their error first makes triage much faster; see failure triage.
Test duration is how long a suite or run takes. Track the total for the pipeline and the slowest individual tests. Long suites get run less often, so duration quietly affects every other metric. When the suite is too slow to run in full, run the riskiest tests first; see risk-based testing.
Metrics that mislead
| Metric | Why it misleads | Use instead |
|---|---|---|
| Number of test cases | More cases is not more coverage; it rewards copy-paste | Requirements verified |
| Bugs found per tester | Rewards splitting bugs and penalises testers on stable areas | Escape rate per release |
| Pass rate on its own | Hides critical failures, small samples and weak passes | Pass rate with completion, critical failures and trusted pass rate |
| Automation percentage | Says nothing about whether automated tests check anything | Trusted pass rate, flaky rate |
| Code coverage as a goal | Easy to raise with tests that assert nothing | Requirements verified, assertion counts |
| Averages across releases | One bad release disappears in the average | The trend per release |
A useful test for any metric: if a team wanted to make it look better without improving quality, how easy would that be? The easier it is, the less weight it should carry.
From metrics to a go/no-go decision
A release decision needs few metrics, agreed before testing starts, each with a threshold. During the go/no-go meeting the team only checks whether the evidence meets them. A practical set:
| Check | Example threshold |
|---|---|
| Runs in scope | At least 1 |
| Execution completion | 100% |
| Pass rate | ≥ 95% |
| Trusted pass rate | ≥ 90% |
| Critical tests failing or blocked | 0 |
| Open critical defects | 0 |
| Requirements verified | ≥ 80%, and none failing |
These are starting points, not industry standards. Tighten them for high-risk or regulated releases. The rule is that the build is ready only when every check passes, and if you ship anyway, you record who decided and why.
Notice what is not in the table: escape rate, defect density and mean time to triage. They are about improving the process over time, so review them in retrospectives, not in the release meeting.
For the full walk-through, see the release readiness checklist and the printable go/no-go checklist template. If you want the checks calculated for you, automated results can be sent from CI as JUnit XML, including from GitHub Actions, and TestPlum works out the verdict from manual and automated results together.
Frequently asked questions
What are the most important QA metrics?
For a release decision: execution completion, pass rate, critical tests failing or blocked, open critical defects and requirements verified, ideally with a trusted pass rate. For improving the process over time: defect escape rate, flaky rate, mean time to triage and test duration.
How do you calculate pass rate?
Divide the number of passed test cases by the number executed, using each case's latest result in scope. Skipped and untested cases belong in execution completion, not in the pass rate.
What is a trusted pass rate?
A pass rate where each pass counts by how much it proves. Passes with no assertions, passes on retry, passes from flaky or since-edited tests and manual passes without an expected result count only partly, so a green suite of weak tests cannot look as good as a solid one.
How do you calculate defect escape rate?
Divide the defects found after release by all defects found for that release, before and after. Use a fixed window, such as 30 days after release, so releases can be compared.
Is defect density a good metric?
Only for spotting where defects cluster and comparing a component with its own past. Size measures such as lines of code are rough, and a low density can mean either good code or weak testing.