Change failure rate is the DORA metric teams fake most often. Not deliberately — it is just genuinely hard. Deployment frequency you can count. Lead time you can measure. But "what percentage of deployments caused a failure in production" requires knowing which deployment caused which failure, and most tools do not know, so they substitute something they can compute and hope the shape is similar.
The proxy, and what it actually measures
The common substitute is a ratio of bugs to delivered work: count the bugs opened in a window, divide by the stories completed, call it a failure rate. It is a reasonable health signal and we still show it. But be clear about what it is — it measures how buggy your work is, not how risky your deployments are. A team that deploys once a month and a team that deploys thirty times a month can produce the identical proxy number while having completely different deployment risk. DORA asks about the second thing.
- The proxy moves when your testing changes, even if deployments are unchanged.
- It cannot tell you which release to roll back, because it never knew about releases.
- It counts bugs found by anyone at any time, including ones that predate the code you shipped.
- Two teams with the same score can have opposite deployment safety.
What a defensible number requires
To attribute a failure to a deployment you need three things, and Azure DevOps already holds all of them. You need the deployment moments — completed pipeline runs, with timestamps. You need the work items each run carried, which Azure DevOps records as the run's associated work items. And you need the bugs, with the date each was opened and the run that later shipped its fix. With those, attribution stops being guesswork and becomes a join.
The evidence standard: confirmed, likely, or neither
The honest part is refusing to convict a deployment on weak evidence. We grade every sampled deployment into one of three states, and only the first two count toward the rate.
- Confirmed failure — a bug was opened during this deployment's window AND a later deployment shipped that bug's fix. Both halves matter: the timing says the bug appeared after this deploy, and the fix link says somebody agreed it was real and repaired it.
- Likely failure — this deployment was followed unusually quickly by a deployment carrying a fix. The pattern of a rushed follow-up is real evidence, but it is circumstantial, so it is labelled separately rather than blended in.
- Clean — no evidence of failure. Not "no failure occurred"; no evidence.
Why "unusually fast" must be relative to your own cadence
This is the detail that decides whether the metric is usable. A fixed threshold — say, any fix deployed within 24 hours convicts the deployment before it — quietly breaks for anyone deploying often. A team that ships ten times a day has every deployment followed within 24 hours, so every bug-fix deploy would accuse its predecessor and the rate would climb toward meaningless. "Unusually fast" only means something against the team's own rhythm. We use half the team's median gap between deployments, capped so a weekly deployer never gets a three-day accusation window. High-frequency teams get a tight window; infrequent deployers get a sane one.
Saying what you could not measure
A rate computed over a sample has to disclose the sample, or it invites a false argument. Bugs that cannot be attributed to any deployment are reported on their own line rather than dropped or spread across deploys. Bugs opened before the sample window starts are excluded and counted. If the deployment sample was capped, that is stated. The point is that when someone challenges the number in a review — and on this metric they will — the answer is on screen, not in somebody's memory of how the tool works.
How this sits next to lead time
Deploy-attributed change failure rate reuses the deployment sample that deploy-linked lead time already fetched, which is why turning it on costs no additional API calls and no additional permission. It also means the two metrics can never disagree about which run shipped what: both resolve a work item's deployment as the earliest sampled run whose linked items include it. If lead time says a story shipped in run 1084, change failure rate is looking at the same run 1084.
Turning it on
Both deploy-linked metrics are opt-in, in Configuration → DORA Metrics, where you pick which pipelines represent a deployment. That choice is yours on purpose: only your team knows which of your pipelines actually deploy and which are CI-only builds, and counting a CI build as a deployment is the fastest way to make every DORA number wrong. Until you opt in, the work-item proxies keep working exactly as before — the attributed card renders alongside the proxy rather than replacing it, so you can see both and decide what to trust.
Related: How to measure DORA metrics in Azure DevOps · Azure DevOps reporting and dashboards