agents-sdk-dotnet-otel

OpenTelemetry for Agents SDK (.NET)

Goal

Configure traces, metrics, and logs for an Agents SDK application, including the SDK's built-in Microsoft.Agents.Core signals and a local Aspire Dashboard. Make the smallest complete change that fits the existing hosting and telemetry architecture.

Use this skill for both AgentApplication and ActivityHandler applications. The SDK telemetry source is the same for both.

Non-negotiable rules

• Inspect the project before editing. Reuse and extend an existing AddOpenTelemetry() registration instead of creating competing providers.
• Register telemetry before builder.Build() and before the app processes agent requests.
• Subscribe to both AgentsTelemetry.SourceName and AgentsTelemetry.Meter; the SDK has no direct OpenTelemetry dependency and publishes through System.Diagnostics.
• Give the application its own stable service.name. Do not use Microsoft.Agents.Core as the service name; that is an instrumentation source, not the application's identity.
• Use OTLP environment variables rather than embedding collector URLs, headers, API keys, or connection strings in source code.
• Never capture message text, prompts, responses, access tokens, authorization headers, cookies, secrets, attachment contents, user names, user IDs, conversation IDs, activity IDs, or arbitrary HTTP headers by default.
• Keep metric attributes low-cardinality. Good dimensions include route type, activity type, channel ID, operation, status, and error type. Do not put per-user, per-conversation, per-message, URL, or exception-message values on metrics.
• Do not add AlwaysOnSampler as a production default. Honor the standard OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG configuration.
• Aspire Dashboard anonymous mode is local-development only. Bind its ports to loopback and do not deploy that configuration.
• Telemetry must never swallow application exceptions or turn failures into successful results.

Workflow

Complete these steps in order.

1. Inspect the application

Find:

• The web or generic-host project (*.csproj) and target framework.
• Program.cs or the host-builder entry point.
• Existing calls to AddOpenTelemetry, ConfigureOpenTelemetry, UseAzureMonitor, AddOtlpExporter, AddApplicationInsightsTelemetry, or custom observability extensions.
• Existing ActivitySource, Meter, counters, histograms, and logging setup.
• Existing OTEL settings in appsettings*.json, launch profiles, environment files, container definitions, or deployment manifests.
• Whether package versions are centrally managed in Directory.Packages.props.

If OpenTelemetry already exists, merge the Agents SDK source and meter into that registration. Do not replace existing exporters or instrumentation without a reason.

2. Add only the required packages

Use the repository's package-management convention. Keep all OpenTelemetry packages on a compatible stable version. When versions are not centrally managed, these commands select the latest stable packages:

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Instrumentation.Runtime

Do not add the console exporter unless the user explicitly wants console telemetry. Do not add Azure Monitor packages unless Azure Monitor export is requested.

3. Configure the providers

Prefer a project-local extension such as AgentOtelExtensions.cs. Adapt the namespace and service name to the project:

using Microsoft.Agents.Core.Telemetry;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

namespace MyAgent;

public static class AgentOtelExtensions
{
    public static TBuilder AddAgentObservability<TBuilder>(
        this TBuilder builder,
        string serviceName,
        string? serviceVersion = null,
        string? applicationTelemetrySource = null)
        where TBuilder : IHostApplicationBuilder
    {
        string resolvedServiceName =
            builder.Configuration["OTEL_SERVICE_NAME"] ?? serviceName;

        var openTelemetry = builder.Services
            .AddOpenTelemetry()
            .ConfigureResource(resource => resource.AddService(
                serviceName: resolvedServiceName,
                serviceVersion: serviceVersion));

        openTelemetry.WithTracing(tracing =>
        {
            tracing
                .AddSource(AgentsTelemetry.SourceName)
                .AddAspNetCoreInstrumentation(options =>
                {
                    options.RecordException = true;
                    options.Filter = context =>
                        !context.Request.Path.StartsWithSegments("/health");
                })
                .AddHttpClientInstrumentation(options =>
                {
                    options.RecordException = true;
                })
                .AddOtlpExporter();

            if (!string.IsNullOrWhiteSpace(applicationTelemetrySource))
            {
                tracing.AddSource(applicationTelemetrySource);
            }
        });

        openTelemetry.WithMetrics(metrics =>
        {
            metrics
                .AddMeter(AgentsTelemetry.SourceName)
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddRuntimeInstrumentation()
                .AddOtlpExporter();

            if (!string.IsNullOrWhiteSpace(applicationTelemetrySource))
            {
                metrics.AddMeter(applicationTelemetrySource);
            }
        });

        builder.Logging.AddOpenTelemetry(logging =>
        {
            logging.IncludeFormattedMessage = true;
            logging.IncludeScopes = true;
            logging.AddOtlpExporter();
        });

        return builder;
    }
}

Notes:

• If the project does not expose /health, keeping the filter is harmless.
• If an existing telemetry extension already adds ASP.NET Core, HTTP client, runtime, or OTLP instrumentation, add only the missing AgentsTelemetry.SourceName source and meter.
• Do not manually add "Microsoft.AspNetCore" or "System.Net.Http" activity source strings when the corresponding instrumentation packages are already configured.
• service.name should identify the application, for example contoso-support-agent. Keep it stable across instances.
• Use service.version for the deployed build/version when one is available.
• Use resource attributes such as deployment.environment.name for stable deployment metadata, not request-specific data.

Call the extension before the agent registrations:

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

builder.AddAgentObservability(
    serviceName: "contoso-support-agent",
    serviceVersion: typeof(Program).Assembly.GetName().Version?.ToString(),
    applicationTelemetrySource: AgentTelemetry.SourceName);

builder.AddAgentDefaults()
    .AddAgent<MyAgent>()
    .AddAgentAuthorization(options => options.AddAgentAspNetAuthentication());

If no custom application telemetry is needed, omit applicationTelemetrySource and do not create AgentTelemetry.

4. Configure OTLP without secrets in source

For a local Aspire Dashboard using OTLP/gRPC:

$env:OTEL_SERVICE_NAME = "contoso-support-agent"
$env:OTEL_EXPORTER_OTLP_ENDPOINT = "http://localhost:4317"
$env:OTEL_EXPORTER_OTLP_PROTOCOL = "grpc"
dotnet run

Standard environment variables to support:

Use user secrets, a secret store, or deployment secrets for OTEL_EXPORTER_OTLP_HEADERS and Azure Monitor connection strings. Never place them in committed appsettings.json or launch profiles.

5. Add custom business telemetry only when useful

The Agents SDK already emits adapter, turn, route, connector, storage, authentication, and authorization signals. Add custom telemetry for application-specific operations, not to duplicate those SDK spans.

Use one shared source name for the application's custom traces and metrics:

using System.Diagnostics;
using System.Diagnostics.Metrics;

namespace MyAgent;

public static class AgentTelemetry
{
    public const string SourceName = "Contoso.SupportAgent";

    public static readonly ActivitySource ActivitySource = new(SourceName);
    public static readonly Meter Meter = new(SourceName);

    public static readonly Counter<long> RouteExecutions =
        Meter.CreateCounter<long>(
            "agent.route.executions",
            unit: "{execution}",
            description: "Number of application route executions.");

    public static readonly Histogram<double> RouteDuration =
        Meter.CreateHistogram<double>(
            "agent.route.duration",
            unit: "ms",
            description: "Application route execution duration.");
}

Instrument a handler with bounded attributes:

using System.Diagnostics;

using Activity? activity =
    AgentTelemetry.ActivitySource.StartActivity("agent.route.execute");
long started = Stopwatch.GetTimestamp();
string status = "ok";

try
{
    await HandleMessageAsync(turnContext, cancellationToken);
}
catch (Exception exception)
{
    status = "error";
    activity?.SetStatus(ActivityStatusCode.Error, exception.Message);
    activity?.AddException(exception);
    throw;
}
finally
{
    TagList tags = new()
    {
        { "route.type", "message" },
        { "status", status }
    };

    AgentTelemetry.RouteExecutions.Add(1, tags);
    AgentTelemetry.RouteDuration.Record(
        Stopwatch.GetElapsedTime(started).TotalMilliseconds,
        tags);
}

If AddException is unavailable for the project's package version, add an exception activity event using the OpenTelemetry semantic attributes. Do not log or tag the prompt, response, or exception message as a metric dimension.

6. Set up the Aspire Dashboard

Copy assets/start-aspire-dashboard.ps1 from this skill into a suitable local development or scripts directory. The script pulls mcr.microsoft.com/dotnet/aspire-dashboard:latest.

Run:

.\start-aspire-dashboard.ps1

Then open http://localhost:18888. The script:

• Binds the UI and OTLP ports to 127.0.0.1.
• Exposes OTLP/gRPC on 4317 and OTLP/HTTP on 4318.
• Enables unsecured access only for local development.
• Runs the dashboard in a disposable Docker container.

Do not start a second container if aspire-dashboard already exists. Inspect or reuse the existing container instead. Stop the generated local dashboard with:

docker stop aspire-dashboard

7. Validate end to end

1. Run the smallest applicable dotnet build.
2. Start the Aspire Dashboard and confirm the container remains running.
3. Start the agent with the OTLP environment variables.
4. Send at least one message through the Agents Playground or configured channel. Metrics are exported periodically and can take one or more export intervals, plus dashboard processing time, to appear. Wait and refresh the dashboard before diagnosing missing metrics.
5. In Aspire Dashboard, confirm:
   • The resource uses the application's service.name.
   • Traces include agents.adapter.process or agents.app.run.
   • Metrics include agents.activities.received, agents.turn.count, or another operation exercised by the test.
   • Application logs appear and correlate with traces when trace context is present.
6. Exercise one failure path when practical and confirm the span is marked Error while the original exception behavior remains intact.

See references/sdk-signals.md for the canonical Agents SDK source, common spans, metrics, and dashboard checks.

Exporter choices

Aspire Dashboard or another OTLP collector

Use OpenTelemetry.Exporter.OpenTelemetryProtocol and AddOtlpExporter(). This is the default for local development and vendor-neutral deployments.

Azure Monitor / Application Insights

When requested, add the stable Azure.Monitor.OpenTelemetry.AspNetCore package and configure UseAzureMonitor() using APPLICATIONINSIGHTS_CONNECTION_STRING from a secret source. Preserve OTLP export only when dual export is intentional. Do not add both exporters accidentally.

Console

Use only for short-lived local diagnosis. Console telemetry is noisy, can expose data in terminal logs, and is not a production exporter.

Troubleshooting

Contributing

If the SDK signals differ from references/sdk-signals.md, the sample no longer builds, or the Aspire setup changes, draft an issue containing the project type, package versions, configuration, expected signal, actual result, and any workaround: https://github.com/microsoft/agents/issues/new