Before you install an analytics extension, check five things: the scopes it requests, where your data goes, what credentials it needs, which accessibility standard it meets, and what the licence terms say. You can answer most of them from the Marketplace listing and the vendor's privacy page.
1. Scopes requested
Every Azure DevOps extension declares its scopes in its manifest. They set the ceiling on what it can read or change.
- Which scopes, exactly? Get the full list if the listing does not show it.
- Read or write? A reporting tool should not need write access to work items. If it asks, find out what it writes and why.
- Does it read code? Agile metrics rarely need source code, so ask what a code scope is for.
- Does each scope map to a feature? A good vendor can say which view uses which scope.
2. Where data goes
Scopes tell you what an extension can read, not where that data ends up.
- Where do the calculations run? In the user's browser, or on the vendor's servers after copying your data? These are very different risks.
- What reaches the vendor? Ask for a specific list. Licence checks, install pings and usage events are normal. What matters is that the list is complete, and whether work-item content is on it.
- Does it use AI? If so, ask what is sent to the model, and whether you can turn it off.
- Is there a telemetry off switch? Who can use it?
- How do integrations work? Do Teams or Slack posts go direct from the browser, or through the vendor?
3. Credentials
Many reporting setups need a personal access token, a service connection, an Azure Function or a database. Each is something you have to store, rotate and audit, and a token tied to one person breaks when that person leaves.
- Does it work with the user's existing Azure DevOps session?
- If an integration needs a credential, where is it stored?
- Is any extra Azure infrastructure required?
4. Accessibility
If your organisation has accessibility obligations, an analytics tool is in scope like any other. Ask which standard the vendor claims, such as WCAG 2.1 AA, and how they check it. An automated suite on a schedule says more than a one-time audit.
5. Licence and trial terms
Read these before anyone starts relying on the dashboards.
- Trial: how long is it, and does it need a card or a form?
- Trial end: are your settings deleted or kept?
- Licence unit: per user, seat or organisation? Does one licence cover several organisations?
- Hosting: Azure DevOps Services, Azure DevOps Server, or both?
- Support: is there a public status page and a written commitment?
Worked example: Agile Analytics
Here is how Agile Analytics by Baytek answers the checklist.
- Scopes: exactly four read-only scopes:
vso.project,vso.work,vso.graphandvso.build. No code scope and no write scope, so it cannot modify work items. - Calculations: analytics views run in the user's browser with their existing session. Work-item content is never sent to a Baytek server.
- What does reach Baytek: licence records, an install and heartbeat ping, usage events you can opt out of, the organisation ID, one administrator contact email, opaque seat identifiers for assigned seats, and Industry Benchmarks data if you opt in.
- AI: when a user asks AGAN AI a question or for a stakeholder summary, the question, a fixed set of setup counts and codes and, on the views that show them, the numbers on that view with, where the view holds them, the same numbers for the previous sprint or period are sent, never the items behind them. An admin can turn AI answers off for the organisation.
- Credentials: no personal access token, Power BI, Analytics views, Azure Function, database or service connection.
- Accessibility: WCAG 2.1 AA, checked by an automated suite that runs nightly.
- Licence: Azure DevOps Services only. When the trial ends, a lock screen appears, nothing is deleted, and entering a key restores access. Each organisation needs its own subscription.
Related: Security · Privacy policy · Azure DevOps analytics with no PAT
Try it
If those answers fit your review, you can run it against your own data. The 30-day free trial starts on install with no credit card, then it is from $300 a year per organisation, with every feature on every plan.
Related: Install instructions · Pricing