azurebackup-telemetry-report

Azure Backup MCP — Weekly Telemetry & Customer Report Generator

Purpose

Generate comprehensive telemetry reports for the Azure Backup MCP toolset by:

1. Querying live production telemetry from Kusto
2. Classifying errors into Customer / Azure Service / MCP Tool Bug
3. Identifying customers via P360/C360 cross-cluster joins (using P360_CustomerName)
4. Correlating with merged PRs and releases
5. Producing an Outlook-compatible HTML report

When to Use

• Weekly cadence (every Monday or Sunday) to produce the status report
• After a release to assess error rate impact
• When triaging production bugs from telemetry data
• When preparing stakeholder updates on Azure Backup MCP health
• Customer adoption reports for any time range (e.g., last 15 days, 30 days)
• When asked "who is using Azure Backup MCP?" or "generate customer report"

Prerequisites

• Azure authentication configured (the Kusto MCP tool needs access to ddazureclients.kusto.windows.net)
• Access to the AzureDevExp database on the Kusto cluster
• Git access to microsoft/mcp repository (for PR/release correlation)

Procedure

Step 1: Query Telemetry Data

Run the following KQL queries against the Kusto cluster using the MCP Kusto tool. All queries use:

• Cluster URI: https://ddazureclients.kusto.windows.net
• Database: AzureDevExp
• Function: getAzureMcpEvents_ToolCalls

1a. Per-Tool Success/Failure with 3-Way Classification (7-day)

This is the primary query. It classifies every call into one of four categories:

• Success: Tool call completed without error
• Customer (4xx): User error — wrong permissions (403), bad input (400), resource not found (404)
• Azure Service: Azure SDK/API/auth layer failed — not an MCP code bug (includes AggregateException, RequestFailedException with 5xx)
• MCP Tool Bug: Our tool code is wrong — FormatException, ArgumentNullException, ArgumentException, etc.

Refer to kql-queries.md for the full query.

1b. Aggregate KPI — This Week vs Last Week

Compare total calls, success rate, customer errors, Azure service errors, and MCP tool bugs between the current week and the previous week. See kql-queries.md.

1c. Error Details

Run the error summary query to get per-tool exception types, messages, and stack traces. Also run the daily error trend query. See kql-queries.md.

1d. Duration Percentiles

Get P50/P95/P99 latency for successful calls. See kql-queries.md.

1e. Custom Dimension Tags

Query azurebackup/VaultType, azurebackup/WorkloadType, azurebackup/DatasourceType, azurebackup/OperationScope dimensions to see if users have upgraded to versions with telemetry tags. Use queries 7, 11, 12, and 13 from kql-queries.md.

Step 2: Gather Git Context

Run these git commands against the microsoft/mcp repository:

# Merged PRs this week
git log upstream/main --oneline --since="<week-start>" -- tools/Azure.Mcp.Tools.AzureBackup/

# Releases shipped
git log upstream/main --oneline --since="<week-start>" --grep="Prepare MCP release"

# Open branches
git branch -a --list "*azurebackup*"

# Changes since last release
git diff <last-release-tag>..upstream/main --stat -- tools/Azure.Mcp.Tools.AzureBackup/

Also count tests on main (checkout upstream/main test files first):

# Runnable unit test count = [Fact] + [InlineData] (each InlineData is a separate test run)
$facts = (Select-String "\[Fact\]" tools/Azure.Mcp.Tools.AzureBackup/tests/Azure.Mcp.Tools.AzureBackup.Tests/**/*.cs | Measure-Object).Count
$inlines = (Select-String "\[InlineData" tools/Azure.Mcp.Tools.AzureBackup/tests/Azure.Mcp.Tools.AzureBackup.Tests/**/*.cs | Measure-Object).Count
# Total runnable = $facts + $inlines

# Live test count — count only [Fact] (each method has [Fact]; [LiveTestOnly] is an
# additional attribute on the same method, so counting both would double-count)
Select-String "\[Fact\]" tools/Azure.Mcp.Tools.AzureBackup/tests/Azure.Mcp.Tools.AzureBackup.Tests/*.cs | Measure-Object

# Registered tools
Select-String "AddCommand" tools/Azure.Mcp.Tools.AzureBackup/src/AzureBackupSetup.cs | Measure-Object

Important: Do NOT count [Fact] + [Theory] as the unit test count — that gives you method count, not runnable test count. Each [InlineData] on a [Theory] is a separate test.

Step 3: Analyze and Classify

Apply the 3-way error classification:

MCP Tool Success Rate = (Total - MCP Tool Bugs) / Total

For each error cluster, determine:

• Is this a new error or seen in previous weeks?
• Is there an existing bug ID (BUG-1 through BUG-8 from the original triage)?
• Has a fix been merged? If so, which PR and release?
• Is the user possibly on an older version (pre-fix)?

Step 4: Generate the Report

Use the report-template.html as a starting point.

The report must include these sections in order:

1. Header — Title, date range, summary line
2. Key Metrics — KPI boxes with week-over-week deltas (Total Calls, MCP Tool Success %, Customer Errors, Azure Service Errors, MCP Tool Bugs, Distinct Users)
3. Releases Shipped — Table of releases with Azure Backup changes
4. Per-Tool Health Table — 3-way classification with MCP % column
5. Error Deep Dive — Analysis of top error patterns
6. Error Hotspots — Table by HTTP status code and classification
7. Merged PRs — Cards for each merged PR with bug references
8. Open PRs — Pending branches
9. Bug Tracker — Updated status of all known bugs
10. Codebase Snapshot — Tool count, test counts, lines changed
11. Action Items — Prioritized next steps

Save the report as azure-backup-telemetry-report-<date>.html (both the styled version and the Outlook-compatible version).

Step 5: Outlook-Compatible Version

For the Outlook version, apply these rules:

• All styles inline (no <style> block or CSS classes)
• Table-based layout instead of flexbox/grid for KPI boxes
• No border-radius, gradients, or CSS pseudo-elements
• Use cellpadding/cellspacing attributes on tables
• Include MSO PixelsPerInch declaration in <head>
• Use border-left on table cells for callout boxes (instead of separate div)

Error Classification Decision Tree

Customer Report Queries: For customer identification, tool adoption, client distribution, and version analysis, refer to kql-customer-queries.md (queries 14–24). These queries use cross-cluster joins to mabprod1 (P360) and icmdataro.centralus (C360) for customer name resolution and external/internal classification.

Key rule: Always use P360_CustomerName (not CustomerName) from the P360 table. The CustomerName field contains generic tenant names (e.g., "Denis" for Prometeia SpA).

Important: MCP bug exception types must be checked before StatusCode. The HandleException base class maps ArgumentNullException (inherits ArgumentException) to HTTP 400, which would incorrectly classify MCP bugs as "Customer" errors if StatusCode is checked first.

Is success == true?
  └─ YES → "Success"
  └─ NO →
      Is ExceptionType FormatException / ArgumentNullException / ArgumentException / InvalidOperationException?
        └─ YES → "MCP Tool Bug"
        └─ NO →
            Is ExceptionType ValidationError?
              └─ YES → "Customer (4xx)" (user omitted required options)
            Is ExceptionType System.UnauthorizedAccessException?
              └─ YES → "Customer (4xx)" (broken credentials / expired token)
            Is ExceptionType Azure.Identity.CredentialUnavailableException?
              └─ YES → "Customer (4xx)" (auth not configured)
            Is StatusCode 400/401/403/404?
              └─ YES → "Customer (4xx)"
              └─ NO →
                  Is ExceptionType Azure.RequestFailedException with 5xx?
                    └─ YES → "Azure Service"
                  Is ExceptionType System.AggregateException?
                    └─ YES → "Azure Service"
                  Otherwise → "Unknown" (investigate)

Known Bug IDs

These are the bugs triaged from the original telemetry analysis (April 2026):

Note: PR #2518 title says "Fix 5 telemetry-triaged bugs (BUG-3,4,5,6,8)" but the actual merged code also includes the BUG-1 fix. However, the #2518 fixes for BUG-1 and BUG-3 were incomplete — confirmed by telemetry on beta.8/beta.9. PR #2621 adds defense-in-depth fixes for both. Always verify the diff, not just the title.

When new errors appear, assign the next sequential ID (NEW-2, NEW-3, etc.) and add to this table.

References

• kql-queries.md — All Kusto queries used in the report
• report-template.html — Outlook-compatible HTML template