Trace an agent run with OpenTelemetry

When a CSR asks why the agent drafted a certificate the way it did, you need the business record: sources, matches, the draft, who approved it. We covered that in logging every agent decision. This post is about the other half, the one engineers need at 8am when 40 COI requests sat in the queue overnight and 6 of them stalled.

That question is not "what did the agent decide", it is "where did the run spend its time, which call failed, and how much did it cost". Decision logs answer the first. Traces answer the second. Mixing them into one table produces something that is bad evidence and bad telemetry at once.

OpenTelemetry now has semantic conventions for generative AI, so the span and attribute names are no longer yours to invent. This tutorial instruments a Python agent run end to end: one trace per work item, child spans for model calls and tool calls, token counts on the spans, and a trace id written into the AMS activity so the two records point at each other.

You need Python 3.9+, an OTLP-compatible backend (we use AWS Distro for OpenTelemetry into CloudWatch; Grafana Tempo, Honeycomb, or Jaeger work the same), and an agent that already does something.

One trace per work item

Pick the unit of work the agency counts and make it the root span. For COI Agent that is one certificate request. For Renewal Radar, one account evaluated. For Service-Request Agent, one inbound message. If your root span is "nightly run", you can tell somebody the batch took 51 minutes and nothing else.

Under the root, spans nest along the shape of the work:

coi.request                        (root, 42s)
├── execute_tool graph.get_message      (0.4s)
├── chat claude-sonnet-4-5              (3.1s)   extract holder requirements
├── execute_tool epic.search_accounts   (1.2s)
├── execute_tool epic.get_policy        (0.9s)
├── chat claude-sonnet-4-5              (6.4s)   check endorsements
├── execute_tool acord25.render         (2.2s)
└── execute_tool epic.create_activity   (1.1s)

The point of the nesting is that a duration is useless without its neighbours. A 42-second request is fine. A 42-second request where 31 seconds is one carrier portal page load tells you what to fix.

Install and start the SDK

pip install opentelemetry-sdk opentelemetry-exporter-otlp

Set the service identity through the environment rather than in code, so the same image runs for two agents:

export OTEL_SERVICE_NAME=coi-agent
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=prod,agency.id=acme"

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("bindmatic.coi")

BatchSpanProcessor matters in a long-running worker: it batches and ships spans on a background thread instead of blocking each model call on a network round trip. In a Lambda, flush the provider before the handler returns or you will lose the tail of every trace.

The root span carries the agency's identifiers

Everything an engineer will search by goes on the root span, and none of it is document content:

def handle_request(req):
    with tracer.start_as_current_span("coi.request") as span:
        span.set_attribute("gen_ai.workflow.name", "coi_draft")
        span.set_attribute("coi.request_id", req.id)
        span.set_attribute("ams.name", "applied_epic")
        span.set_attribute("ams.account_id", req.account_id or "")
        span.set_attribute("coi.source", "mailbox")
        try:
            result = run(req)
        except Exception:
            span.record_exception(...)
            raise
        span.set_attribute("coi.outcome", result.outcome)   # drafted | escalated | no_match
        span.set_attribute("coi.confidence", result.confidence)
        return result

Custom attributes go under your own prefix (coi., ams.). The gen_ai.* names come from the convention. Do not invent your own spelling of something the spec already names, and do not put a spec name on a value that does not match its definition.

Model calls: use the gen_ai convention

The GenAI semantic conventions define span names as {gen_ai.operation.name} {gen_ai.request.model}, so chat claude-sonnet-4-5. The attributes you want on every model span:

with tracer.start_as_current_span(f"chat {model}") as span:
    span.set_attribute("gen_ai.operation.name", "chat")
    span.set_attribute("gen_ai.provider.name", "aws.bedrock")
    span.set_attribute("gen_ai.request.model", model)
    span.set_attribute("gen_ai.request.max_tokens", 4096)
    span.set_attribute("gen_ai.request.temperature", 0)

    resp = client.messages.create(...)

    span.set_attribute("gen_ai.response.model", resp.model)
    span.set_attribute("gen_ai.response.id", resp.id)
    span.set_attribute("gen_ai.response.finish_reasons", [resp.stop_reason])
    span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output_tokens)
    span.set_attribute("gen_ai.usage.cache_read.input_tokens",
                       getattr(resp.usage, "cache_read_input_tokens", 0) or 0)

Two notes on the convention as it stands. gen_ai.provider.name replaced the older gen_ai.system; if your vendor library still emits gen_ai.system, that is why. And the GenAI conventions are still marked experimental, so pin your instrumentation library version and expect an attribute rename at some upgrade. That is cheaper than a private schema nobody else's tooling can read.

Token attributes are what make cost queryable per request rather than per month. It is the same counting the per-task spend cap needs, so emit once and read it twice.

Tool calls, including the ones that are not models

Every AMS read, carrier portal fetch, mailbox call, and PDF render gets a span. The convention names the operation execute_tool:

from contextlib import contextmanager

@contextmanager
def tool_span(name, **attrs):
    with tracer.start_as_current_span(f"execute_tool {name}") as span:
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", name)
        span.set_attribute("gen_ai.tool.type", "function")
        for k, v in attrs.items():
            span.set_attribute(k, v)
        yield span

with tool_span("epic.search_accounts", **{"ams.name": "applied_epic"}) as span:
    hits = epic.search_accounts(name=holder_name)
    span.set_attribute("ams.match_count", len(hits))
    span.set_attribute("ams.match_score", hits[0].score if hits else 0.0)

ams.match_count and ams.match_score are the two attributes we reach for most often in practice. An agent that suddenly escalates everything is almost always matching zero or matching three, and the trace shows which before anyone opens a document.

What must not go on a span

Spans leave the agency's account and land in a monitoring backend with different access rules than the AMS. So:

• No document bytes, no message bodies, no prompt or completion text. The convention has opt-in fields for message content (gen_ai.input.messages, gen_ai.output.messages) and off is the right default on a book of business containing EINs, loss history, and driver names. Vendor auto-instrumentation sometimes captures content by default; check the flag, do not assume.
• No insured names or holder names. Use the AMS account id. An engineer who needs the name can look it up with the right permissions, which is the point.
• No credentials, and no full portal URLs with session tokens in the query string.

Keep prompts and drafts in the agency's own decision log, where the redaction pass already applies and retention is under the agency's control.

Sample on outcome, not on volume

Head sampling throws away 90% of traces before anything is known about them, which reliably discards the 6 stalled requests and keeps 34 boring ones. Use the OpenTelemetry Collector's tail sampling processor instead: buffer the spans of a trace, then decide.

processors:
  tail_sampling:
    decision_wait: 30s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: escalations
        type: string_attribute
        string_attribute: { key: coi.outcome, values: [escalated, no_match] }
      - name: slow
        type: latency
        latency: { threshold_ms: 30000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

Keep every failure, every escalation, every slow run, and a tenth of the clean ones for a baseline. On agent volumes, that is a small bill and a complete record of everything interesting. Set decision_wait longer than your slowest root span, or long traces get cut in half.

One thing tail sampling does not cover: a run that waits on a human. Do not hold a trace open across a three-day approval pause. End the drafting trace at the approval gate, and start a new trace when the decision comes back, carrying the same coi.request_id so a query joins them.

Write the trace id into the AMS

This is the step that makes traces usable by someone who is not looking at the monitoring tool. When the agent writes its activity, include the trace id:

ctx = trace.get_current_span().get_span_context()
trace_id = format(ctx.trace_id, "032x")

epic.create_activity(
    account_id=req.account_id,
    code="COI-DRAFT",
    description=f"COI drafted for review. Trace {trace_id}.",
)

Now the path runs both ways. An account manager who sees an odd activity can hand the engineer an id. An engineer who finds a bad trace knows exactly which account and policy it touched. No timestamp archaeology.

Three metrics, then stop

Traces explain one run. Metrics tell you whether today is worse than last Tuesday. Three are enough to start:

• Requests by outcome (drafted, escalated, no_match) as a counter. Escalation rate is your accuracy proxy between formal reviews.
• Root span duration as a histogram, watched at p95. Averages hide the stalls.
• Input and output tokens as counters, which the same span attributes already give you.

Alert on the escalation rate rather than on latency. A slow agent annoys a CSR; an agent whose escalation rate quietly dropped from 12% to 2% is either better or it stopped checking something, and you want to know which within a day, not at the next review.

Limits

Tracing tells you what the agent did and how long it took. It does not tell you whether the answer was right. That is what scoring a silent run and accuracy review on real cases are for; the two are complementary and neither substitutes for the other. Traces also are not an E&O record: retention in a monitoring backend is typically 15 to 30 days, and the record you would produce for a carrier needs to live as long as the file does, in the AMS.

We instrument this way inside the agency's own single-tenant AWS account, so the telemetry belongs to the agency along with the source code. If you are running agents against a live book and cannot answer "what happened on that request", tell us your AMS and the workflow and a senior engineer will reply within one business day.