Executives read a short set of delivery metrics: a forecast with a confidence level, the items at risk, the direction of delivery over time, and a single line on delivery performance. Azure DevOps holds the data for all four. Agile Analytics, an Azure DevOps extension, packs them into a one-page Exec Report that you can share as a PDF, a copied summary or copied email text. This post is about choosing what to say, not just what to chart.
What leaders ask
Leaders rarely ask for a cycle time scatterplot. They ask four questions, usually in the same meeting.
- When will it be done, and how sure are we?
- What is likely to slip, and what needs a decision from me?
- Are we getting better or worse than a few sprints ago?
- Are we shipping safely?
Every metric on a leadership page should answer one of these. If a chart does not, it belongs in the team’s own review, not in the update. The Exec Report is described in the product as a one-page leadership summary of forecast, risks, delivery trend and DORA, with per-team detail after it. That structure matches the four questions, so you are not deciding what to cut each time. You are checking that each section says something true and useful this week.
Forecast with confidence levels
“When will it be done?” is the question that most often produces a bad answer. A single date sounds precise but hides the uncertainty, and leaders end up treating it as a promise. A better answer is a range with a confidence level. The Monte Carlo view runs 10,000 simulations on your team’s historical throughput. It answers two questions: how many items will we finish by a given date, and when will we be done with what is left. Outcomes come at 95%, 85%, 70% and 50% confidence, which you will also see described as P50, P85 and P95 dates.
The Exec Report puts the forecast in one sentence of the same kind, phrased as the percentage chance of finishing at least a number of items in the next few weeks. When you write the forecast up for leadership, a few habits help:
- Give two dates, not one. A P50 date is the “could well happen” case and a P85 date is the “plan around this” case. The gap between them tells leaders how much uncertainty is in the work.
- Say which confidence level you committed to. If a date matters to a customer or a contract, a higher confidence level is the honest choice. If it is an internal target, a lower one may be fine.
- Explain the basis once. “This comes from how many items the team actually finished in past sprints” is enough. Leaders do not need a lesson on simulation.
Related: How to choose the right forecast confidence level · Azure DevOps Monte Carlo forecasting
Risks and aging
The second question is about what might go wrong. The most reliable early warning in a flow system is work that has been in progress too long. The Aging & Blocked view plots every in-flight item by days in flow against the team’s p50 and p85, ranked by urgency. Blocked items are shown too, and the state lanes follow the team’s board order, so the picture matches what the team sees on its own board. An item that has passed the team’s p85 is taking longer than most of the similar work the team has finished, which is a reasonable trigger for a conversation. The risks block of the Exec Report counts open items older than the p85 cycle time and shows bugs opened and closed over a recent window.
For the leadership page, keep the risk section short. Name the few items that are old or blocked, say what is blocking them, and state what you need from the reader, if anything. A list of ten aging items teaches leaders to skim. A list of two with a clear ask gets action. If your team sets WIP limits, the WIP Monitor tracks work in progress against per-team limits and can send Teams or Slack webhook alerts. That is useful for the team day to day, and you can mention in the update that limits are being watched without adding another chart.
Delivery trend
The third question is about direction. A single sprint result is noisy, and leaders learn little from it. A trend across several sprints is much more useful. The Exec Report includes a delivery trend section, and the views behind it are where you go to check the story before you share it:
- Delivery Signals shows commitment reliability, carryover trend and time to start across past sprints.
- Flow Metrics shows throughput, lead time and WIP trends over rolling windows.
- Process Behavior draws XmR charts with upper and lower control limits, which help separate normal variation from a real change.
That last one matters more than it sounds. If throughput dipped in one sprint and the point sits inside the control limits, the honest message is “within normal variation, no action needed”. Leaders appreciate being told when not to worry, and it keeps real signals credible when you do raise them. When you describe the trend, use plain words, such as “carryover has gone down over the last few sprints” or “lead time is steady”. Avoid attaching causes you have not checked. If you know the reason, say it. If you do not, say that too.
Delivery performance in one line
DORA metrics are deployment frequency, lead time for changes, change failure rate and time to restore. The DORA Metrics view in Agile Analytics computes them from work items and pipelines, and it uses the vso.build scope to read pipeline data. The Exec Report carries a shorter delivery performance block, which shows deployments per week, median lead time and build failure rate. For a leadership page the right size is one line. Something like “Delivery to production is steady and failures are being recovered from quickly” says what a leader needs, provided it is what the numbers show.
One caution: because these figures are computed from work items and pipelines, they reflect how your team uses both. Before the first time you share them upward, compare them with what the team knows about its own releases. If they match, you have a baseline. If they do not, fix the data conventions first, then share. The in-product glossary, How These Metrics Work, explains each metric.
Sharing without a BI tool
Many teams build executive reporting in a separate BI tool, then maintain it. That is a reasonable choice for some organisations, and the Analytics versus Power BI comparison lays out the differences. If your goal is a clear one-page update, though, you can skip that step. Agile Analytics needs no Power BI, no Analytics views, no personal access token, no Azure Function, no database and no service connection. The Exec Report can be shared in three ways:
- As a PDF. The report opens your browser’s print dialog, with the leadership summary on page one and team detail after it, ready to save as a PDF.
- As email text. Copy email text puts a subject line and the summary on your clipboard, ready to paste into your own email to a stakeholder.
- As a copied summary, for pasting into a document, wiki or chat message.
Related: Azure DevOps Analytics vs Power BI · Azure DevOps reporting
The report is built entirely in your browser and sends nothing to Baytek. More broadly, every analytics view runs in the browser against the Azure DevOps REST API, using your existing session, and work item content such as titles, sprint contents and comments is never sent to a Baytek server. The extension requests four read-only scopes (vso.project, vso.work, vso.graph and vso.build) and cannot modify work items. That is a short answer to give a security reviewer who asks where the data goes. Two limits to state plainly: Agile Analytics supports Azure DevOps Services (cloud) only, not Azure DevOps Server, and whoever builds the report needs Project Contributor access or higher.
Try it on your own project
Agile Analytics starts a 30-day free trial automatically on install, with no credit card and no sign-up form. During the trial there are no seats, so everyone in the organisation has full access and you can open the Exec Report against your own project data. If you later want to compare plans, they are on the pricing page.
Related: Install instructions · Pricing