Series: QA Leadership · Article 5 of 9
Quarterly review. Beautiful numbers on the slide. Escaped Rate: 1.8%. A downward trend for three quarters. Everyone happy. Until someone asked: "So which release exactly was the problem in Q3?"
PM"Good result. But I remember that payments-module incident in August - was that our release?"
QAopens the per-release table...
QA"v3.4 - 7 escaped bugs. The other 7 releases combined: 4."
PM"So one release accounted for 64% of the quarter's problems. And the report didn't say a word about it."
The overall Escaped Rate doesn’t lie. It just hides. One catastrophic release dissolves into the average of the other seven. And that’s exactly why you need Escaped Bugs per Release - the metric that looks at every release individually.
Escaped Rate vs Escaped per Release - what each one sees
These are not two versions of the same metric. They are two completely different levels of vision. And you need both - but with full awareness of what each one tells you, and what it doesn’t.
Overall Escaped Rate
Rate = Total escaped ÷ (Escaped + Pre-release) × 100%
Says: how effective we are overall
Quarterly trend, industry benchmark, an argument for the board. Good for high-level reporting and comparing quarters.
⚠️ Doesn't show which releases were risky. A single spike gets lost in the average.
Escaped per Release
EpR = Escaped bugs attributed to a specific release
Says: which release was risky
Incident diagnosis, pattern hunting, a causes conversation with your EM. Good for identifying problems and preventive action.
✓ Every release gets its own score. A spike is visible immediately.
The same quarter - two completely different diagnoses
The table below shows how aggregation hides the real picture. Overall Escaped Rate: 1.8% - looks good. But broken down per release, a completely different story emerges.
| Release | Escaped bugs | EpR | Verdict |
|---|
| v3.1 | 0 | 0.0 | Elite |
| v3.2 | 1 | 0.2 | Elite |
| v3.3 | 0 | 0.0 | Elite |
| v3.4 SPIKE | 7 | 2.8 | Alarm |
| v3.5 | 1 | 0.3 | Good |
| v3.6 | 0 | 0.0 | Elite |
| v3.7 | 1 | 0.2 | Elite |
| v3.8 | 1 | 0.3 | Good |
| Q3 total | 11 | 1.4 avg | Hidden catastrophe |
The overall Rate for the whole quarter: 1.8% - “looks good”. And one release accounted for 64% of the problems.
How much is too much - the thresholds
EpR is a close cousin of the industry’s most recognizable stability metric - Change Failure Rate from the DORA research. CFR asks: what percentage of deployments cause problems in production? EpR asks: how many problems exactly did each deployment bring? In DORA terminology, a release with at least one escaped bug is a “failed change”.
It’s worth knowing the current context: the DORA 2025 report moved away from the classic Elite/High/Medium/Low levels toward seven team archetypes, and the bar for top performance was raised - the best teams keep CFR in the 0-2% range, which only about 17% of organizations achieve. The term “elite” is still widely used across the industry as shorthand - and that’s how I use it below - but be aware that formally DORA speaks a different language today.
Thresholds for Escaped Bugs per Release, calibrated to those standards:
<0.5
Elite
High-performing teams. The vast majority of releases ship with zero escaped bugs. Sporadic single incidents.
0.5-1.5
Good
A mature process. An occasional escaped bug, quickly detected and fixed. No recurring pattern.
1.5-3.0
Needs attention
Visible process problems. Customers regularly feel the impact. Root-cause analysis needed.
>3.0
Alarm signal
Every such release is an incident that requires a post-mortem. Immediate process intervention.
For teams in an enterprise environment with regular releases every 1-2 weeks - the target is below 0.5 escaped bugs per release. Above 1.5 is a signal that something is systemically broken.
The spike you see right away
The Escaped per Release chart immediately reveals what the aggregate statistic hides. One glance - and you know which release needs an investigation.
Escaped Bugs per Release - per-release view
v3.4 visible instantly. The overall average: 1.4 - it would look innocent.
Spike: v3.4 = 2.8
Escaped per Release
Elite threshold (<0.5)
Alarm threshold (3.0)
Escaped per Release vs overall Escaped Rate - the same story
Rate aggregates and smooths. EpR reveals anomalies. You need both perspectives.
two perspectives
Escaped per Release (count)
Overall Escaped Rate (%)
Investigation framework
5 questions you ask after every spike
A spike by itself is not a problem - it's a signal. The real problem starts when the spike gets logged as an "anomaly" and nobody looks for the cause. Here are five questions that always lead to an answer.
Three rules before you start, taken straight from SRE culture. First - the investigation is blameless. We ask "how did the system allow this bug?", never "who is at fault?". A team afraid of blame stops telling the truth - and without truth there is no diagnosis. Second - we look for contributing factors, not a single root cause. Complex systems fail through several interacting conditions at once; "one cause" is almost always a simplification that loses the rest of the picture. Third - we run the investigation within 48 hours of detecting the spike, while the context is still fresh in the team's heads.
1
What was different about this release compared to the previous ones?
A spike is rarely accidental. Something changed - in the code, the process, the team or the environment. The first question is always: what was different?
New module or technologyTeam composition changeSchedule changeExternal integration
2
What type of bugs dominated - functional, configuration, integration?
The type breakdown (from article 3) points to where to look for the cause. A dominance of configuration bugs suggests an environment problem, not code quality.
Code → dev processConfiguration → DevOpsIntegration → architectureRegression → tests
3
Did QA have enough time and access to the environment?
A spike under deadline pressure is not a coincidence. If the testing window shrank by 40% - and 40% more bugs reached production - that's a simple correlation worth showing your PM.
Testing time vs the previous releaseTest environment stabilityStaging availabilityCode delivery delays
4
Were the bugs in areas covered by automated tests?
If the escaped bugs lived in modules with 90% automated coverage - the problem is the quality of the tests, not their number. If they were in uncovered areas - that's a signal to fill the gap.
Coverage map vs bug locationLast test update for that moduleFlaky tests in that area
5
Have you seen a similar spike before - and what helped then?
Your EpR history is your biggest asset. If the spike in v2.8 had the same profile as the spike in v3.4 - and a code review checkpoint solved it back then - you have a ready-made corrective action.
Previous spikes in your EpR historyWhat was done after the last spikeWhether the corrective action was kept up
Interactive spike detector
Enter the Escaped per Release values from your last 6 releases. The detector automatically flags spikes and tells you which release needs an investigation.
EpR spike detector
Enter the number of escaped bugs per release - the spike threshold uses the IQR method (Tukey): Q3 + 1.5 × IQR
From diagnosis to prevention - how to use spikes
A spike identified and investigated is half the battle. The other half is turning the findings into actions that make sure history doesn’t repeat itself.
How to roll it out from scratch
1
Collect historical EpR - at least the last 6 releases
Attribute every escaped bug (from Jira, monitoring, support) to the specific release that introduced it - not to the date it was detected. That's the key difference. A bug found in week 3 after the release still belongs to that release.
Even approximate data going 2-3 months back gives you a first trend and identifies historical spikes.
2
Set the spike threshold with the IQR method - the statistical standard
The industry standard for outlier detection is Tukey's method based on the interquartile range: threshold = Q3 + 1.5 × IQR, where Q3 is the 75th percentile and IQR is the difference between the 75th and 25th percentiles of your EpR data. The method is robust against the spikes themselves - unlike mean-based thresholds, which a single spike can heavily inflate.
With few releases (fewer than 8-10 data points) treat the result as indicative - and update the threshold quarterly, because as the team matures, the distribution tightens and the threshold becomes more demanding.
3
Introduce a blameless post-mortem for every spike - 30 minutes, within 48h
It doesn't have to be a multi-hour meeting. A 30-minute retrospective with the 5 questions from the framework above - run within 48 hours of detection, while the context is fresh. The overriding rule: blameless - we analyze the system and the process, not people. If you need a deeper causal technique, the classic Five Whys (keep asking "why" until you reach a systemic cause) works great in this format.
The post-mortem output is a list of contributing factors plus action items - each with an owner and a deadline. The document goes into the team archive: when the next spike appears, you start by checking whether the cause profile is already known.
4
Flag "high-risk" releases ahead of time
After a few months of data you'll start seeing patterns: new technologies → higher risk. Onboarding a developer → higher risk. Deadline pressure → higher risk. Use those patterns for a pre-mortem before the release - don't wait for a spike to react.
Four preventive actions - when to use each
🔍
Pre-mortem before a risky release
Ask "what could go wrong?" before the deployment, not after. Especially for releases with new integrations or architecture changes.
When: before every "big" release
📋
Release risk checklist
A simple list of questions to fill in before each release: new developer? new integration? shortened testing window? Every "yes" raises the risk category.
When: every release, 5 minutes
🐤
Canary deployment for high risk
Release to 5-10% of traffic before the full rollout. EpR for that window is an early signal - you can roll back before the problem reaches all customers.
When: EpR historically > 1.5 for similar releases
⏱️
Post-deploy monitoring window
The first 2 hours after deployment - active monitoring with lower alert thresholds. A large share of EpR spikes is detected exactly in this window.
When: always, automatically
EpR in the business conversation
Sprint Review
"This release ended with 0 escaped bugs - elite-level EpR. The previous v3.4 was a spike with an EpR of 2.8. We ran a post-mortem and introduced a code review checkpoint. The result is already visible in this release."
1:1 with EM
"The overall Escaped Rate looks good - 1.8%. But per release you can see that one release accounted for 64% of the quarter's problems. I have a hypothesis about the cause and a proposal for a corrective action."
Board
"We introduced per-release EpR monitoring. In Q4 we identified two spikes, ran a post-mortem for both and implemented corrective actions. Average EpR dropped from 1.4 to 0.4 - elite level. All Q4 releases below the 0.5 threshold."
What EpR gives you that the overall Rate doesn’t
✓ EpR gives you
- Identification of the specific risky release - not an averaged statistic
- A starting point for a causal investigation (5 questions)
- Historical patterns for predicting risky releases
- A measurable effect of corrective actions per release
- An argument in the PM conversation about testing time and conditions
✗ EpR does not replace
- The overall Escaped Rate - for the quarterly trend and industry benchmarks
- DDR - for evaluating the effectiveness of the testing process
- Issues per Release - for the maturity of the code entering testing
- Number of Releases - without release-count context, EpR loses its scale
A spike is not a problem - it's a signal. The problem starts when the spike gets logged as an "anomaly" and nobody looks for the cause. EpR gives you the tool to change that.
In the next article
Article six covers Number of Releases - the metric most teams ignore completely. And without it, every other metric in this series loses its scale and comparability.
You’ll learn why 3 escaped bugs across 2 releases is a crisis, while 3 escaped bugs across 15 releases is a success - and how to use this metric as the common denominator for the whole series.
Series: QA metrics the business wants to hear
- 01
Diagnosis, three pillars, five metrics, the QA → KPI mapping model
- 02
Formula, thresholds, historical data, seasonality, pitfalls
- 03
Taxonomy, data collection, the cost of each type, how to report
- 04
Rollout from scratch, the link to the development process, the EM conversation
- 05
Escaped Bugs per Release you are here
Spike detection, the investigation framework, preventive actions
- 06
Number of Releases - the context metric
Why 3 bugs with 2 releases is a disaster, and with 15 - a success
- 07
Release Confidence Score step by step
Three calculation models, rollout, concrete examples from practice
- 08
Storytelling with metrics - building a narrative
How to turn a table of numbers into a business argument
- 09
3 anti-patterns that destroy QA credibility
Too many metrics, no context, jargon - and how to avoid each