telemetry-foundations

Telemetry Foundations

Overview

A shared vocabulary for observability across HVE Core agents. This skill describes what telemetry data exists and how it is named, not which SDK or vendor to use. Agents producing planning artifacts (ADRs, PRDs, security/RAI plans, code-review reports) and agents producing user-facing application code reference this skill so that downstream pipelines (traces, metrics, logs) speak a consistent OpenTelemetry-aligned language.

When to Apply

Apply this skill in the following situations:

• Any agent producing user-facing application code that emits spans, metrics, or structured logs.
• Architecture Decision Records that touch observability, monitoring, or audit logging.
• Code-review reports that flag telemetry gaps, inconsistent span naming, or unbounded metric cardinality.
• Security or Responsible AI plans that cite audit logs, traceability, or evidence chains.
• Product or business requirement documents that specify success metrics expressed as service telemetry.

Core Principles

The vocabulary in this skill follows five principles:

• Declarative, not prescriptive. Define the names and shapes; leave the choice of SDK, exporter, and backend to the implementing team.
• OpenTelemetry-aligned. Trace, metric, and log models follow the OTel data model so artifacts remain portable.
• Semantic conventions first. Where an OTel semantic convention exists for a domain (HTTP, RPC, database, messaging, GenAI, FaaS), prefer it over a bespoke attribute.
• PII by denylist. Treat PII as default-deny via the denylist in references/pii-denylist.md; any field listed there requires an explicit redaction strategy before it can be emitted.
• Vendor-agnostic. Avoid coupling vocabulary to a single backend; OTLP is the assumed wire protocol.

Trace Vocabulary

Spans describe a unit of work and its causal relationship to other work.

Span kinds:

• server - inbound request handled by this service.
• client - outbound request issued by this service.
• producer - asynchronous message published to a queue or topic.
• consumer - asynchronous message received from a queue or topic.
• internal - in-process operation with no remote peer.

Required resource-scoped attributes on every span:

• service.name
• service.version
• deployment.environment

Span naming pattern: <verb>.<resource> using lowercase dot-separated tokens. The verb describes the operation (get, create, publish, consume, query); the resource describes the target entity (order, customer, payment.intent). Examples: get.order, publish.order.created, query.customer.by_email.

For domains covered by OTel semantic conventions (HTTP, RPC, database, messaging, GenAI, FaaS), use the convention's span-naming guidance instead of the generic pattern above.

Metric Vocabulary

Metrics describe aggregate measurements over time.

Instrument types:

• counter - monotonic, additive (request count, bytes sent).
• up-down-counter - non-monotonic, additive (queue depth, active connections).
• histogram - distribution of values (request duration, payload size).
• gauge - last-sampled value, non-additive (CPU temperature, memory in use).
• observable-counter, observable-up-down-counter, observable-gauge - async variants polled by the SDK.

Unit conventions follow UCUM. Examples: s (seconds), ms (milliseconds), By (bytes), 1 (dimensionless count). Express durations as histograms in seconds (s) by default to align with OTel HTTP semantic conventions.

Metric naming pattern: <domain>.<entity>.<measure> using lowercase dot-separated tokens. Examples: http.server.request.duration, db.client.connections.usage, messaging.publish.duration.

Cardinality discipline: every attribute attached to a metric multiplies the time-series count. Bound high-cardinality dimensions (user ID, request ID, free-form strings) at the source or move them to exemplars and traces.

Log Vocabulary

Structured logs carry discrete events with severity and context.

Severity levels (OTel log data model):

• TRACE - fine-grained diagnostic detail, off by default.
• DEBUG - diagnostic detail useful during development.
• INFO - normal operational events.
• WARN - unexpected condition that does not block the operation.
• ERROR - operation failed; the caller likely saw a failure.
• FATAL - process is going to terminate.

Recommended structured fields on every log record:

• timestamp (ISO-8601, UTC).
• severity_text and severity_number.
• body (the human-readable message; structured data goes in attributes).
• attributes.* (typed key-value pairs scoped to this event).
• resource.* (inherited from the producing service).

Trace correlation: when a log record is emitted within an active span, inject trace_id and span_id so traces and logs join cleanly downstream. Logging libraries that integrate with the OTel context propagator do this automatically.

PII Handling

Personally Identifiable Information is handled by denylist. The authoritative list lives in references/pii-denylist.md. Treat any field on that list as default-deny: do not emit it as a span attribute, metric dimension, or log field without an explicit redaction strategy.

Redaction patterns:

• Hash - one-way hash (SHA-256, optionally truncated) for fields that must remain joinable across events but should not be human-readable.
• Drop - omit the field entirely from telemetry.
• Tokenize - replace with an opaque token resolvable only through a separate, access-controlled store.

Identifier convention: where a stable user reference is needed in telemetry, use user.id populated with an opaque hash of the canonical user identifier, never the raw email, phone, or external account ID.

When introducing a new attribute that could contain PII, add it to the denylist first and choose a redaction strategy before emitting it.

Sampling and Cost

Sampling controls the volume of telemetry shipped to downstream collectors.

Sampling strategies:

• Head-based - decision made at span start, propagated through the trace. Low cost, simple, but cannot bias toward late-discovered properties (such as errors).
• Tail-based - decision made after a trace completes, typically in a collector. Higher cost, allows policies such as "keep all error traces" and "keep slow traces".

Defaults:

• Use a parent-based sampler so child spans inherit the parent's sampling decision and traces remain whole.
• When tail-based sampling is available, bias toward keeping error traces and a representative sample of successful traces.

Metric and log sampling: metrics are pre-aggregated and rarely sampled; logs are typically rate-limited per severity rather than sampled.

Resource Attributes

Resource attributes describe the entity producing the telemetry and are attached to every span, metric, and log record automatically by the SDK.

Required:

• service.name
• service.version
• deployment.environment
• telemetry.sdk.name
• telemetry.sdk.language
• telemetry.sdk.version

Recommended when applicable:

• cloud.* (cloud provider, region, account ID).
• k8s.* (cluster name, namespace, pod name).
• host.* (hostname, architecture, OS type).

Follow the OTel Resource Semantic Conventions for canonical attribute names.

Decision Tree

Use this quick-select when choosing whether and how to instrument:

1. Is this user-facing or part of a user-visible flow? If no, prefer DEBUG logs and skip span/metric emission unless needed for capacity planning.
2. Is the cardinality of the proposed attributes bounded? If no, move the unbounded field to a log attribute or trace exemplar rather than a metric dimension.
3. Does the data contain or derive from a field in references/pii-denylist.md? If yes, apply a redaction strategy before emitting.
4. Does the operation cross a service boundary (network, queue, process)? If yes, emit a span with the matching server, client, producer, or consumer kind and propagate context.
5. Is the operation high-volume? If yes, rely on parent-based sampling and (where available) tail-based policies; do not disable instrumentation outright.
6. Does an OpenTelemetry semantic convention cover this domain? If yes, use its attribute names and span-naming guidance; if no, follow the naming patterns in this skill and propose a new entry in references/proposed-additions.md.

References

Authoritative external sources:

• OpenTelemetry Semantic Conventions v1.41.0
• W3C Trace Context
• OpenTelemetry Protocol (OTLP) Specification
• OpenTelemetry Logs Data Model
• OpenTelemetry FaaS Semantic Conventions

Portions adapted from OpenTelemetry Semantic Conventions, (C) OpenTelemetry Authors, licensed under CC BY 4.0.

Internal:

• references/pii-denylist.md - authoritative PII denylist with redaction strategies.
• references/proposed-additions.md - intake for new vocabulary proposals.