Study notes · two weeks · 3–4 h a week
Agent Observability
When an agent misbehaves in production, the trace is the only witness. This path covers the shared vocabulary for agent telemetry, then wires it into Azure so every run can be replayed.
The core material
| Resource | Role | Use it for |
|---|---|---|
| OpenTelemetry GenAI conventions, explained | Mental model | Agent and tool spans, gen_ai.* attributes, token and agent metrics, current status |
| Foundry agent tracing | Enterprise platform | OTel traces for Foundry agents, stored in Application Insights |
| VS Code: monitor agents with OpenTelemetry | Local lab | Export Copilot agent telemetry to the Aspire dashboard or Jaeger on your laptop |
- The conventions are still in Development. Names can change between releases, and in June 2026 they moved to a new semantic-conventions-genai repository. Pin your instrumentation version.
- No Educative pick here. I didn't find an Educative course that teaches agent tracing in depth; the free material above is better.
01 / Week 1
The GenAI telemetry vocabulary
Learn the span shapes, attributes and metrics that every agent framework is converging on.
| Signal | Name | What it captures |
|---|---|---|
| Span | invoke_agent | One agent run; nested for sub-agents and agent-to-agent calls |
| Span | execute_tool | One tool call, including calls made over MCP |
| Span | chat / embeddings / retrieval | Model inference and retrieval steps |
| Span | invoke_workflow, plan | Multi-agent workflows and planning phases |
| Metric | gen_ai.client.token.usage | Tokens by type, operation and model; the basis for cost |
| Metric | gen_ai.invoke_agent.tool_calls | Tool calls per run; compare with inference calls to spot loops |
Source: Dash0 explainer
Read (~2 h)
- OpenTelemetry GenAI semantic conventions explained
- OTel GenAI agent spans: now a pointer to the new repository; check it for the latest names
- Foundry agent tracing concepts: includes the multi-agent conventions Microsoft and Cisco Outshift contributed
Privacy default: Prompt and response text are opt-in events, not span attributes, because they often hold personal data. Keep secrets out of tool arguments.
CheckpointSketch the trace tree for a supervisor agent that calls two sub-agents, one of which calls an MCP tool.
02 / Week 2
Wire it up locally and on Azure
See traces on your laptop first, then send the same signals to Application Insights through Foundry.
Do (~4 h)
- Monitor agents with OpenTelemetry (VS Code): run the Aspire dashboard or Jaeger and watch real agent spans arrive
- Set up tracing in Foundry: connect Application Insights and instrument your framework
- Flow Framework events and traces: every turn, model call, tool and schedule is an event; OTLP traces optional
- Build one dashboard: runs per hour, p95 latency per tool, tokens and cost per agent, and tool calls per run.
Pair with evals: Traces feed evals. Sample production traces into the regression set from the Agent Evaluation path.
CheckpointFrom one bad answer in production, you can open its trace and name the step that went wrong in under five minutes.
Capstone
Instrument one real agent end to end. Done when every line holds:
- Every run produces an invoke_agent span with nested execute_tool spans
- Token usage and cost per agent are on a dashboard
- Prompt content capture is off by default and redaction is in place
- An alert fires when tool calls per run spike, a sign of looping
- Traces land in Application Insights for the Azure deployment
- I can link any eval failure to its trace