Agent Monitoring and Evaluation

Agent Monitoring and Evaluation in Foundry

Introduction: The Necessity of Oversight in Agentic Systems

When we talk about building "agents"—software systems that use Large Language Models (LLMs) to perform tasks, make decisions, and interact with external tools—we are moving beyond simple request-response patterns. Unlike traditional software, where logic is explicitly hard-coded and deterministic, agents are probabilistic. They choose their own paths, interpret ambiguous instructions, and determine which tools to execute based on the context they are provided. This autonomy is powerful, but it introduces a significant challenge: how do we ensure these systems are performing as expected, behaving safely, and delivering actual value?

Agent monitoring and evaluation (often referred to as "evals") are the bedrock of reliable agentic infrastructure. Monitoring is the continuous observation of an agent’s behavior in production—tracking latency, cost, error rates, and tool usage patterns. Evaluation is the process of testing an agent against a set of benchmarks to ensure that changes to the system (like updating a prompt or swapping a model) result in improvements rather than regressions. Without these two pillars, you are essentially flying blind, hoping that your agent’s "reasoning" remains aligned with your business logic.

In the context of Foundry, monitoring and evaluation are not just "nice-to-haves"; they are fundamental to the lifecycle of the application. As your agent matures, you will inevitably need to adjust its system prompt, change its tool definitions, or upgrade to a newer model version. Without a rigorous evaluation framework, you will have no way of knowing if these changes have introduced subtle flaws, such as hallucinated tool calls or incorrect data extraction, until a user reports a failure. This lesson will guide you through the architecture of a monitoring and evaluation strategy tailored for agentic workflows.

The Core Components of Agent Monitoring

To monitor an agent effectively, you need to capture data at every stage of the agent’s execution loop. In a standard Foundry-based agent setup, an agent typically follows a cycle: perception (receiving input), reasoning (planning/thought process), action (calling tools), and reflection (evaluating the tool output). If you only log the final answer, you lose visibility into the "why" behind the agent's decisions.

1. Observability: Capturing the Execution Trace

The execution trace is a sequential log of every step the agent takes. In Foundry, this involves capturing the full conversation history, the intermediate "thoughts" generated by the LLM, the specific arguments passed to tools, and the raw responses returned by those tools. You should treat this trace as the primary source of truth for debugging.

• Input Context: What was the original user query? What system instructions were active?
• Reasoning Steps: What did the model "think" it needed to do before taking an action?
• Tool Execution: Which tool was selected? What were the parameters? Did the tool succeed or fail?
• Latency Metrics: How long did each step take? Where is the bottleneck?
• Token Usage: How many input and output tokens were consumed at each step?

2. Monitoring Key Performance Indicators (KPIs)

Once you have the traces, you need to aggregate them into actionable metrics. These metrics help you understand the health of your agent at a glance.

• Success Rate: The percentage of tasks completed without errors or user intervention.
• Cost Per Task: The cumulative cost of all LLM calls required to reach a final answer.
• Tool Accuracy: How often does the agent select the correct tool for a given task?
• Hallucination Rate: The frequency with which the agent references non-existent data or misinterprets tool output.
• Retry Frequency: How often does the agent have to re-evaluate its plan due to a failed tool call or an error from the model?

Callout: Monitoring vs. Logging While many developers confuse the two, there is a distinct difference. Logging is the act of recording events as they happen—essentially a digital diary. Monitoring is the proactive analysis of those logs to identify patterns, set alerts for anomalies, and ensure that the system is operating within defined performance boundaries. Logging tells you what happened; monitoring tells you if what happened was acceptable.

Implementing Evaluation Strategies

Evaluation is the process of grading your agent’s output. Since agents often generate creative or unstructured responses, traditional unit testing (which looks for exact string matches) is rarely sufficient. Instead, you need a multi-layered evaluation strategy.

1. Deterministic Evaluation (Unit Tests)

For specific components, such as data extraction or tool parameter formatting, you can use deterministic tests. If your agent is tasked with parsing a date from a natural language string, you should have a suite of test cases with known "ground truth" outputs.

# Example of a deterministic test for a tool argument parser
def test_date_parser():
    input_text = "Schedule the meeting for next Tuesday at 2 PM"
    expected_output = "2023-10-24 14:00"
    
    # Run the agent's parsing logic
    actual_output = agent.parse_date(input_text)
    
    assert actual_output == expected_output, f"Expected {expected_output}, got {actual_output}"

2. Model-Based Evaluation (LLM-as-a-Judge)

This is the industry standard for evaluating complex agentic behavior. You use a "Judge" model (usually a more capable model, like GPT-4o or Claude 3.5 Sonnet) to grade the output of your agent. You provide the judge with a rubric, the user query, and the agent's response, and ask it to assign a score.

• Relevance: Did the agent answer the user's question?
• Factuality: Is the information provided supported by the tool outputs?
• Safety: Does the response violate any content guidelines?
• Efficiency: Did the agent reach the answer in an optimal number of steps?

Note: When using an LLM as a judge, always include a "chain of thought" requirement in your prompt to the judge. By forcing the model to explain why it gave a specific score, you make the evaluation process transparent and easier to audit.

3. Human-in-the-Loop (HITL) Evaluation

No matter how advanced your automated evaluation is, human review remains essential for high-stakes agents. You should build a feedback loop where users can "thumbs up" or "thumbs down" an agent’s response. These human-labeled examples become your "Golden Dataset," which you can use for regression testing in the future.

Best Practices for Agent Monitoring

Monitoring an agent is a continuous process that evolves as the agent learns and as the underlying models change. Below are the best practices to ensure your monitoring framework remains effective.

• Establish a Golden Dataset: Create a set of 50–100 representative queries that cover the most common use cases for your agent. Every time you change a system prompt or update the agent's logic, run these queries against the new version and compare the results to your "Golden" results.
• Track Tool Failure Modes: Not all errors are the same. A tool failing because of a timeout is different from a tool failing because the agent passed invalid arguments. Categorize your errors to understand whether the issue lies in your infrastructure or the agent's reasoning.
• Set Up Alerts for Cost Spikes: Agentic loops can sometimes get stuck in a "thought loop," where the agent calls tools repeatedly without making progress, burning through tokens. Set up alerts for high token consumption or unusually long execution times for a single user request.
• Version Control Everything: Treat your prompts, tool configurations, and model versions as code. Use tags to identify which version of the agent produced a specific trace. This is crucial for reproducing bugs that happen in production.
• Analyze "Tool Selection" Bias: If your agent has access to five tools, it might prefer one over the others due to how the prompt is structured. Periodically review which tools are being called and ensure the distribution matches your expectations.

Callout: The "Golden Dataset" Concept A Golden Dataset is your most valuable asset in agent development. It is a curated collection of inputs, expected tool calls, and expected final answers. Think of it as the unit tests of the agent world. By maintaining this dataset, you can perform "A/B testing" on your prompts and models, ensuring that a change intended to improve performance doesn't inadvertently break existing functionality.

Common Pitfalls and How to Avoid Them

Even with a robust monitoring system, developers often fall into traps that can obscure the true performance of their agents. Avoiding these common mistakes is key to maintaining a production-grade system.

1. Over-Reliance on Aggregate Metrics

Relying solely on averages—like "average response time"—can be dangerous. An agent might be fast on 90% of requests but consistently fail or time out on the 10% that involve complex, multi-step tasks. Always look at the distribution of your metrics (e.g., P95 and P99 latency) rather than just the mean.

2. Neglecting the "Failure Path"

Most developers test for the "happy path" where the agent succeeds on the first try. However, the most critical part of an agent's logic is how it handles failure. What happens when a tool returns an error? Does the agent try to fix the arguments and retry? Does it inform the user? You must explicitly monitor and test the agent's behavior during these error-recovery cycles.

3. Failing to Monitor Model Drift

Models are updated frequently by providers. A system prompt that worked perfectly with a specific version of a model might behave differently after a silent update. You should pin your model versions (e.g., gpt-4o-2024-05-13 instead of just gpt-4o) and perform regression testing whenever you plan to migrate to a newer model version.

4. Ignoring Context Window Saturation

As an agent interacts with tools and performs multiple steps, the conversation history grows. If you don't monitor the token count, the agent might eventually hit its context limit, leading to truncated responses or forgotten instructions. Monitor the token usage per turn to ensure your context management strategy (e.g., summarization or history pruning) is working.

Step-by-Step: Setting Up a Basic Monitoring Pipeline in Foundry

To implement a monitoring pipeline, you need to hook into the agent's execution lifecycle. The following steps outline how to capture and evaluate agent performance within a Foundry-based architecture.

Step 1: Instrumenting the Agent

You need to wrap your agent’s execution function with a telemetry provider. This provider will record the start time, the input, the tool calls, and the final output.

import time
import uuid

def execute_with_monitoring(agent, user_query):
    request_id = str(uuid.uuid4())
    start_time = time.time()
    
    try:
        # Execute the agent
        response = agent.run(user_query)
        
        # Log success
        log_to_database(
            request_id=request_id,
            query=user_query,
            response=response,
            duration=time.time() - start_time,
            status="success"
        )
        return response
    except Exception as e:
        # Log failure
        log_to_database(
            request_id=request_id,
            query=user_query,
            error=str(e),
            status="failed"
        )
        raise e

Step 2: Capturing Intermediate Steps

To get full visibility, you must capture the "thought" process. If you are using a framework, look for "hooks" or "callbacks" that trigger every time a tool is called.

# Pseudo-code for a callback-based monitoring approach
def on_tool_call(tool_name, arguments):
    # Record that a tool was called
    db.save_event("tool_called", {
        "tool": tool_name,
        "args": arguments,
        "timestamp": time.time()
    })

# Register the callback with the agent
agent.set_tool_callback(on_tool_call)

Step 3: Running Automated Evals

Create a dedicated evaluation script that runs your Golden Dataset against your current agent deployment.

def run_evaluations(agent, golden_dataset):
    results = []
    for entry in golden_dataset:
        prediction = agent.run(entry['input'])
        score = evaluate_with_llm(entry['expected'], prediction)
        results.append({"input": entry['input'], "score": score})
    
    # Calculate average score
    return sum(r['score'] for r in results) / len(results)

Quick Reference: Evaluation Metrics

Advanced Concepts: Proactive Guardrails

While monitoring tells you what happened, guardrails prevent bad things from happening. In a mature agentic system, you should implement input and output validation layers.

• Input Guardrails: Check the user query for prohibited topics, prompt injection attempts, or malformed data before it reaches the agent.
• Tool Guardrails: Validate the arguments passed to a tool before the tool is executed. For example, if a tool accepts a date, ensure the model isn't trying to pass a string like "next year" that the tool can't parse.
• Output Guardrails: Review the final response for PII (Personally Identifiable Information), brand tone compliance, or factual consistency before showing it to the user.

Tip: Start small with your evaluation suite. Don't try to build a perfect automated evaluation system on day one. Begin by logging all your agent conversations and manually reviewing them once a week. Once you identify common patterns of failure, you can start writing automated tests to catch those specific issues.

Addressing Common Questions

How do I handle non-deterministic behavior?

Agents will never be 100% deterministic. Instead of testing for exact equality, test for semantic similarity. Use embedding-based similarity scores or LLM-as-a-judge to determine if the agent's answer is "good enough" given the intent of the user.

How do I store and query agent traces?

For small projects, a simple JSON-based logging system works fine. For production systems, you should use a dedicated observability platform or a time-series database (like Prometheus or specialized LLM monitoring tools) that allows you to slice and dice your data by user, model version, or time range.

How often should I run evaluations?

Run your basic regression suite on every code push (CI/CD). Run your full, comprehensive evaluation (including human review) whenever you make significant changes to the system prompt or the agent's architecture.

Summary: Key Takeaways for Agentic Success

Building agents in Foundry requires a shift in mindset from traditional software development. You are managing a probabilistic system, and your primary responsibility is to provide the guardrails and visibility necessary to keep that system safe and effective.

1. Visibility is Paramount: You cannot improve what you cannot see. Ensure you are logging the entire agentic flow, including intermediate thoughts and tool calls, not just the final result.
2. Evaluation is a Lifecycle: Evaluation isn't a one-time step. It is a continuous loop that starts with a Golden Dataset and persists through production as you monitor for regressions and model drift.
3. Use AI to Monitor AI: Leverage more capable "Judge" models to evaluate the performance of your agents. This is the only way to scale your testing beyond manual human review.
4. Prioritize the Failure Path: The quality of an agent is defined by how it behaves when things go wrong. Spend as much time testing error handling and recovery as you do testing successful outcomes.
5. Version Everything: Treat your prompts, tool definitions, and agent logic as versioned assets. This allows you to roll back changes when an update causes unexpected behavior in production.
6. Focus on Metrics that Matter: Avoid vanity metrics. Focus on task success, tool precision, and cost, ensuring that your agent is delivering value while staying within your resource constraints.
7. Implement Guardrails: Don't wait for a failure to happen. Use input and output validation layers to catch issues before they reach the user, effectively turning your monitoring into a proactive safety system.

By adhering to these principles, you will be able to build agents that are not only capable and autonomous but also reliable and predictable enough to trust with real-world business processes. Monitoring and evaluation are the difference between a brittle prototype and a resilient, production-ready agentic solution.