See what your AI agent did and why

An AI agent can return the right answer and still take the wrong path. It may call an unnecessary tool, expose sensitive data to telemetry, retry an expensive step, or act with more authority than the task requires.

Observability gives a team evidence about that path. It does not make an agent safe by itself; it supports testing, investigation, and human control.

Observe the whole run

Logs that record only the final response leave out the steps most likely to explain a failure. Capture the inputs, model calls, tool calls, results, latency, token use, errors, retries, and final outcome that belong to one run.

Use a trace identifier to connect those events. Redact personal data, credentials, and sensitive tool arguments before the telemetry leaves the application boundary.

Microsoft Foundry uses OpenTelemetry-based tracing for supported agent scenarios. Its agent tracing overview also notes that some agent and workflow capabilities remain in preview. Check the status for the framework and region you plan to use.

What production work demands of observability

Monitor actions, not only outcomes

Record which tool an agent selected, the arguments it sent, the result it received, and the action it took next. Define which actions require approval and which are read-only.

An alert should point to a useful condition, such as repeated tool failure, an unexpected write, a latency threshold, or a change in task completion.

Trace decisions with context

A trace should show the information available at each step without pretending to expose private model reasoning. Keep the prompt version, retrieved sources, policy result, tool response, and workflow state needed to reproduce the run.

This lets a reviewer ask a concrete question: which input or tool result changed the path?

Evaluate repeatable tasks

Build an evaluation set from representative tasks, edge cases, and known failures. Measure criteria that matter for the use case, such as groundedness, tool accuracy, task completion, safety, latency, or cost.

Run the same checks before release and after a meaningful change. Microsoft documents how evaluations can act as a quality gate in a delivery pipeline.

Put policy checks in the workflow

Access control, approval, data-loss prevention, and business policy belong at the point where an action is proposed. A prompt that says “follow policy” is not an enforcement boundary.

Record the policy version and result so an auditor can see why the workflow allowed, blocked, or escalated an action.

Connect observability to release management

Version agent instructions, tools, data connections, policies, and evaluation sets together. A deployment should identify which version produced a run and which checks it passed.

In production, compare new traces with the pre-release baseline. Route failures to an owner who can pause an action or revert the change.

Decide what happens when evidence is incomplete

Observability is useful only when the team agrees how to respond. Define:

• which events trigger an alert
• who can inspect sensitive traces
• when a person must approve an action
• how long telemetry is retained
• which failure pauses the workflow
• what evidence is required before release resumes

The Microsoft Foundry agent development lifecycle places tracing, evaluation, publishing, and monitoring in one iterative process. Your operating model still decides the thresholds and owners.

The Odie Bot case study shows a conversational service in a public-data setting. It is related delivery evidence, not proof that every observability control above was used in that project.