Evaluating Models and Flows

Module: Implement Generative AI Solutions

Lesson: Evaluating Models and Flows in Microsoft Foundry

Introduction: Why Evaluation Matters

In the landscape of modern artificial intelligence, the transition from a prototype to a production-ready application is defined by one critical phase: evaluation. Building a generative AI solution using Microsoft Foundry (Azure AI Foundry) is straightforward when you are simply wiring up a model to an API endpoint. However, ensuring that the model behaves predictably, safely, and accurately across thousands of real-world inputs is a different challenge entirely. Evaluation is the process of measuring the quality, safety, and performance of your generative AI flows before they reach your users.

Without a rigorous evaluation strategy, you are essentially flying blind. You might find that your application works perfectly for a handful of test cases you hand-picked, only to have it hallucinate, leak sensitive data, or provide irrelevant answers when exposed to the messy, unpredictable nature of actual user data. Evaluation allows you to quantify the "goodness" of your model’s output, compare different models against one another, and track how your system improves—or regresses—as you iterate on your prompts and logic.

This lesson explores how to approach evaluation within the Microsoft Foundry ecosystem. We will move beyond simple "vibes-based" testing and look at systematic ways to use metrics, automated evaluators, and human-in-the-loop workflows to create reliable AI solutions.

The Anatomy of an Evaluation Workflow

An evaluation workflow in Microsoft Foundry typically consists of three primary components: the target (the flow or model you are testing), the test dataset (a set of questions and expected answers), and the evaluators (the metrics that score the output).

1. The Target: This is your generative AI flow. It could be a simple prompt-response pair, a RAG (Retrieval-Augmented Generation) pipeline, or a complex agentic workflow that calls multiple tools.
2. The Test Dataset: This is a JSONL or CSV file containing input prompts and, ideally, "ground truth" or reference data. Ground truth provides a benchmark for what a correct answer looks like.
3. The Evaluators: These are the scoring mechanisms. They can be mathematical (e.g., word overlap), model-based (using a "judge" LLM to grade output), or safety-based (checking for hate speech or PII).

Callout: Deterministic vs. Probabilistic Evaluation It is important to distinguish between deterministic and probabilistic evaluation. Deterministic evaluation uses fixed rules to compare outputs, such as checking for the presence of a specific keyword or calculating string similarity. Probabilistic evaluation, which is more common in generative AI, uses another "judge" model to assess qualitative attributes like "relevance," "coherence," or "groundedness." Because generative models are non-deterministic, probabilistic evaluation is often the only way to measure nuanced performance.

Step-by-Step: Running Your First Evaluation

To begin evaluating a flow in Azure AI Foundry, you must first define your test data. Let's assume you are building a customer support bot for a hardware retailer.

1. Preparing the Data

Your dataset should be representative of the queries your bot will actually handle. For RAG applications, you need the user query, the retrieved context (documents), and the model’s generated response.

{"query": "How do I reset my router?", "context": "To reset the Model X router, press and hold the button on the back for 10 seconds.", "response": "Press the button on the back for 10 seconds to reset your router."}
{"query": "What is the return policy?", "context": "Returns are accepted within 30 days of purchase with a receipt.", "response": "You can return items within 30 days if you have your receipt."}

2. Selecting Your Evaluators

Microsoft Foundry provides a suite of built-in evaluators. You don't need to write custom logic for every metric. The most common metrics you should start with include:

• Groundedness: Measures whether the response is supported by the provided context (prevents hallucinations).
• Relevance: Measures how well the response addresses the user’s query.
• Coherence: Measures how logically the response flows.
• Fluency: Measures the grammatical correctness of the response.

3. Executing the Evaluation

You can trigger an evaluation run directly from the Azure AI Foundry portal or via the SDK. Using the SDK provides better version control and integration into your CI/CD pipelines.

from azure.ai.evaluation import evaluate

# Define the path to your test dataset
data_path = "./test_data.jsonl"

# Define the evaluators
evaluators = {
    "groundedness": GroundednessEvaluator(),
    "relevance": RelevanceEvaluator()
}

# Run the evaluation
result = evaluate(
    data=data_path,
    evaluators=evaluators,
    target=my_flow_endpoint # The URI of your deployed flow
)

Tip: Use a "Judge" Model When running model-based evaluators, ensure you are using a high-performing model (like GPT-4o) as the evaluator, even if your application is using a smaller or cheaper model. The evaluator must be "smarter" than the model being evaluated to provide meaningful feedback.

Deep Dive: Evaluating RAG Pipelines

RAG (Retrieval-Augmented Generation) is the most common generative AI pattern, but it is also the hardest to evaluate because there are two failure points: the retrieval process (did we find the right documents?) and the generation process (did the model synthesize the answer correctly?).

The RAG Triad

The RAG Triad is a framework for evaluating these systems:

1. Context Relevance: Did the retriever find the right information?
2. Groundedness: Did the LLM use that information correctly?
3. Answer Relevance: Did the LLM answer the user's question?

If you have poor performance, the RAG Triad helps you diagnose the root cause. If your groundedness score is low, your prompt or model is at fault. If your context relevance score is low, your search index or embedding model needs tuning.

Common Pitfalls in RAG Evaluation

Many developers make the mistake of evaluating only the final output. If the answer is wrong, they don't know if the search failed or the model hallucinated. Always log the intermediate steps—the specific chunks retrieved from your vector database—so you can evaluate them individually.

Comparison of Evaluation Methods

Best Practices for Robust Evaluation

1. Build a "Golden Dataset"

A golden dataset is a collection of 50–100 high-quality question-answer pairs that represent the "ideal" behavior of your application. Treat this dataset like a unit test. Every time you change a prompt or swap a model, run your flow against this golden dataset to ensure you haven't introduced regressions.

2. Monitor for Safety

Evaluation isn't just about utility; it's about safety. Always include automated safety evaluators to check for:

• Jailbreak attempts: Does the user try to bypass safety filters?
• PII leakage: Does the model reveal sensitive user information?
• Tone/Sentiment: Is the model being too informal or aggressive?

3. Use Automated Pipelines

Do not run evaluations manually. Integrate your evaluation scripts into your GitHub Actions or Azure DevOps pipelines. When a developer pushes a code change to the prompt library, the pipeline should automatically trigger an evaluation run and output a report. If the score drops below a pre-defined threshold, the build should fail.

Warning: The "Average" Trap Relying solely on average scores can hide significant problems. If your model scores 90% relevance on average, you might be ignoring the 10% of cases where the model gives dangerous or completely wrong answers. Always inspect the outliers—the low-scoring results—to understand the edge cases where your system fails.

Handling Common Mistakes

Mistake 1: Evaluating with the Same Data Used in Training

If you use your test data for fine-tuning or prompt engineering, you are essentially "leaking" the answers to the model. This leads to overfitting, where the model performs well on your test set but fails in the real world. Always keep a hold-out test set that the model has never seen.

Mistake 2: Ignoring Latency and Cost

Evaluation often focuses on accuracy, but in production, latency is a feature. A highly accurate model that takes 20 seconds to respond is often worse than a slightly less accurate model that responds in 2 seconds. Include latency metrics in your evaluation dashboard to ensure your "better" model is actually viable for production.

Mistake 3: Over-reliance on Single Metrics

No single metric captures the entirety of a model's performance. A model might be highly "fluent" but completely "un-grounded." Use a composite score or a dashboard view that shows multiple metrics simultaneously to get a holistic view of performance.

Advanced: Custom Evaluators

Sometimes, the built-in metrics in Azure AI Foundry won't satisfy your specific requirements. For instance, if you are building a legal document analyzer, you might need an evaluator that checks if the model cited specific statutes correctly.

You can create a custom evaluator by defining a prompt that acts as a grader.

# Custom evaluator example
def legal_citation_evaluator(response, context):
    # Logic to check if citation exists in response
    # This could be a regex or a secondary LLM call
    if "statute_reference" not in response:
        return 0
    return 1

Integrating this into your evaluate() call allows you to extend the standard metrics to fit your domain-specific needs. This is particularly useful in highly regulated industries like finance, healthcare, or law.

The Role of Human-in-the-Loop (HITL)

While automated evaluation is essential for scale, it should never fully replace human judgment. Automated evaluators can be biased or make mistakes themselves.

• Sampling: Regularly sample 5–10% of your automated evaluation results for human review.
• Disagreement Analysis: When the human reviewer disagrees with the automated judge, analyze why. Does the judge need a better prompt? Is the human being too subjective?
• Feedback Loops: Use the results of your human reviews to refine your golden dataset, effectively creating a "self-improving" evaluation system.

Integrating Evaluation into the Development Lifecycle

To maintain quality over time, consider the following lifecycle:

1. Experimentation: Developers tweak prompts and model parameters. They use local, small-scale evaluations.
2. Validation: Once a change looks promising, it is run against the full golden dataset using the CI/CD pipeline.
3. Deployment: If the validation passes, the change is deployed to a staging environment.
4. Monitoring: In production, you continue to collect data. Periodically, you pull this data, label it, and add it back to your golden dataset to ensure your test suite evolves with the real-world usage of your app.

Summary Checklist for Success

• Define Objectives: What does "success" look like for your specific application?
• Curate Data: Build a diverse, representative golden dataset.
• Automate: Integrate evaluation into your CI/CD pipeline.
• Triangulate: Use multiple metrics (RAG Triad, safety, latency).
• Review: Periodically perform manual audits of automated scores.
• Iterate: Use evaluation results to inform prompt engineering and retrieval strategy.

Frequently Asked Questions

Q: How many test cases do I need in my golden dataset? A: Start with 50. As you gain more production traffic, aim for 200-500. Quality is more important than quantity; 50 well-chosen, diverse examples are better than 1,000 repetitive ones.

Q: What if my evaluator model is too expensive? A: Use a smaller, cheaper model (like GPT-4o-mini or a fine-tuned smaller model) for the evaluator. As long as it is consistently applied, you can measure relative improvements even if the absolute score is slightly different from a larger model.

Q: Can I evaluate a flow that doesn't have a structured output? A: Yes, but it is harder. You may need to add a "formatting" step to your flow that forces the output into JSON or a specific structure before it reaches the evaluator.

Q: How do I evaluate "creativity" or "tone"? A: These are subjective. Use a rubric-based evaluator where you provide the LLM with a 1-5 scale and specific definitions for what a "1" vs. a "5" looks like in your context.

Key Takeaways

1. Evaluation is a Continuous Process: It is not a one-time activity at the end of a project. It must be embedded into the development lifecycle to catch regressions early.
2. Use the RAG Triad: For retrieval-based systems, always evaluate context relevance, groundedness, and answer relevance separately to identify where your pipeline is failing.
3. Automate Everything: Manual testing is slow and prone to human error. Use the Azure AI Foundry SDK to automate your evaluation runs within your deployment pipelines.
4. Prioritize Ground Truth: A golden dataset of high-quality examples is your most valuable asset. Invest time in creating and maintaining it.
5. Don't Rely on Averages: Look at the outliers and the failures. A high average score can mask critical safety or accuracy issues that will hurt your users.
6. Combine Automated and Human Review: Use automated judges for scale and human review for calibration and edge-case validation.
7. Monitor Production: Evaluation doesn't stop at deployment. Use production monitoring to capture new edge cases, which then feed back into your golden dataset for future testing.

By following these practices, you transform your development process from a guessing game into a rigorous engineering discipline. The goal is not to achieve a "perfect" score, but to create a system where you understand exactly how your model performs, where it is likely to fail, and how to improve it systematically. Microsoft Foundry provides the tools to do this; your responsibility is to build the culture of testing and verification around those tools.