Document Processing Models

Managing Environments: Configure AI Builder - Document Processing Models

Introduction: The Power of Intelligent Document Processing

In the modern digital workplace, organizations are flooded with documents. From invoices and tax forms to identity cards and purchase orders, the sheer volume of unstructured data contained within these physical or digital files represents a significant bottleneck for business processes. Manually extracting data from these documents is not only tedious and prone to human error, but it also scales poorly as a company grows. This is where AI Builder’s Document Processing models come into play.

Document Processing is a feature within the Microsoft Power Platform that uses machine learning to identify, extract, and structure information from documents. By training a model to "read" specific document types, you can automate data entry into systems like Dataverse, SharePoint, or Dynamics 365. This transition from manual data handling to automated intelligence allows your team to focus on higher-value decision-making rather than repetitive administrative tasks. Understanding how to configure these models is a foundational skill for any administrator or developer working within the Power Platform ecosystem.

Understanding Document Processing Models

At its core, a Document Processing model is a specialized AI engine that you train using your own business documents. Unlike prebuilt models—which are designed for generic tasks like reading business cards or passports—a custom document processing model is tailored to your unique forms. Whether you have a proprietary invoice template or a specific legal contract layout, the model learns the visual structure and the specific fields you need to extract.

How the Model Learns

The learning process relies on a technique called "supervised learning." You provide the AI with a set of sample documents (training data) and manually label the fields you want it to find. Over time, the model builds a mathematical understanding of where these fields typically appear, the labels associated with them, and the format of the data (e.g., currency, dates, or addresses). Once the training threshold is met, the model can process new, unseen documents and extract the data with high accuracy.

Callout: Custom vs. Prebuilt Models It is important to distinguish between custom and prebuilt models. Prebuilt models are "off-the-shelf" solutions for common document types. They require no training data but are limited to specific structures. Custom Document Processing models require a representative set of your actual documents but provide the flexibility to capture any field relevant to your business, regardless of how unique your document layout might be.

Preparing for Configuration: Prerequisites and Environment Setup

Before you start building your first model, you need to ensure your environment is configured correctly. AI Builder is a feature that requires specific licensing and environment permissions. If you are working in a production environment, ensure you have the "Environment Maker" or "System Administrator" role assigned to your user account. Furthermore, you must have an AI Builder capacity allocation, which is typically managed at the tenant level.

Data Collection Strategy

The quality of your model is directly proportional to the quality of your training data. Do not simply grab random documents from your file server. You should select a representative sample that covers the variations you expect to see in production. For example, if your invoices sometimes arrive with a logo on the left and other times with a logo on the right, ensure your training set includes both variations.

• Diversity: Include documents from different vendors or departments if they follow the same general structure.
• Quality: Ensure the document images are clear and readable. While AI Builder can handle some noise, heavily blurred or distorted scans will result in poor extraction performance.
• Quantity: Aim for at least 5-10 documents per "collection" or layout. While you can technically train with fewer, more examples generally lead to a more stable and accurate model.

Tip: The Power of Collections If you have multiple types of layouts for the same document category (e.g., invoices from Vendor A vs. Vendor B), use "Collections" within the AI Builder interface. A collection acts as a grouping mechanism that allows the model to learn multiple, distinct layouts simultaneously while keeping the training data organized.

Step-by-Step Configuration Guide

Configuring a Document Processing model involves a structured workflow within the Power Apps maker portal. Follow these steps to build your first model.

Step 1: Create the Model

1. Log in to the Power Apps maker portal.
2. On the left navigation pane, select AI Hub and then AI models.
3. Select Document processing from the list of available models.
4. Provide a name for your model and click Create.

Step 2: Define Information to Extract

In this phase, you define the fields, tables, and checkboxes you need to extract.

• Fields: These are single values, such as an Invoice Number, Total Amount, or Customer Name.
• Tables: These are repeating data structures, such as the line items on an invoice (e.g., Description, Quantity, Unit Price).
• Checkboxes: These are binary indicators (Selected/Not Selected) for things like "Terms and Conditions Accepted."

Step 3: Upload and Label Documents

Once your fields are defined, you must upload your sample documents.

1. Create a new collection.
2. Upload your documents.
3. Once uploaded, click on each document to open the labeling interface.
4. Highlight the text on the document and tag it with the corresponding field you defined in Step 2.
5. Repeat this for all documents in your collection.

Step 4: Train and Publish

After labeling, click the Train button. The AI Builder engine will process your labeled data. Once the training completes, you will receive a performance score. If the score is satisfactory, click Publish. Your model is now ready to be used in Power Automate flows or Power Apps.

Handling Complex Data: Tables and Repeating Items

One of the most common requirements for document processing is extracting line items from tables. This is more complex than extracting a single field because the number of rows can vary from one document to the next. AI Builder handles this by allowing you to define a "Table" field type.

When labeling a table, you define the columns (e.g., Item Description, Quantity, Price). You then draw a bounding box around the entire table area and tag each cell appropriately. The model learns not just the text inside the cells, but the structure of the grid itself. This allows the model to distinguish between a "Subtotal" row at the bottom of the table and the actual line items.

Warning: Complexity Limits While AI Builder is powerful, it has limits regarding extremely complex, multi-page tables that span across different pages with varying headers. If your documents have highly inconsistent table structures, consider using a combination of AI Builder for header data and a custom script or a different tool for complex, multi-page table parsing.

Integrating with Power Automate

The true value of a Document Processing model is unlocked when you integrate it into an automated workflow. Power Automate provides a native connector for AI Builder that allows you to trigger processes automatically when a document is uploaded to a location like SharePoint or OneDrive.

Example Workflow: Automated Invoice Processing

1. Trigger: A file is created in a SharePoint folder named "Invoices."
2. Action: Use the "Predict" action from the AI Builder connector. Select your published Document Processing model.
3. Input: Pass the file content from the SharePoint trigger to the Predict action.
4. Action: Use the extracted data to create a record in Dataverse or send an email notification to an approver.

// Example of accessing extracted data in a Power Automate expression
// This assumes the output of the Predict action is stored in 'Predict_Results'
{
  "InvoiceNumber": "@{outputs('Predict')?['body/predictionOutput/fields/InvoiceNumber/value']}",
  "TotalAmount": "@{outputs('Predict')?['body/predictionOutput/fields/TotalAmount/value']}",
  "LineItems": "@{outputs('Predict')?['body/predictionOutput/tables/LineItems']}"
}

This snippet demonstrates how you access the JSON output generated by the model. The predictionOutput object contains all the mapped fields and tables you defined during the training phase.

Best Practices for Model Maintenance

A model is not a "set it and forget it" asset. As your business changes, so will your documents. You must implement a strategy for maintaining your models to ensure they remain accurate over time.

• Performance Monitoring: Regularly check the performance metrics provided in the AI Builder dashboard. If accuracy begins to dip, it is usually a sign that your documents have changed (e.g., a new invoice template was introduced) and the model needs to be retrained.
• Retraining: Keep your original training dataset. When you add new document types, add the new samples to your existing set and retrain the model. This prevents "catastrophic forgetting," where the model loses its ability to read the old formats while learning the new ones.
• Version Control: Give your models clear names and version numbers (e.g., "Invoice_Processor_v1.2"). This makes it easier to roll back if a newly trained model performs worse than the previous version.

Common Pitfalls and How to Avoid Them

Even experienced developers encounter issues when working with AI Builder. Awareness of these common pitfalls can save hours of troubleshooting.

1. Inconsistent Data Labeling

The most frequent cause of low accuracy is inconsistent labeling. If you tag "Invoice Total" on one document but forget to tag it on another, the model will struggle to learn the pattern. Ensure that every field you define is tagged consistently across all training documents.

2. Overloading the Model

Trying to extract too many fields can confuse the model, especially if those fields are visually similar. If you find your model is struggling, try reducing the number of fields to only the most critical ones, or split your model into two separate models if the documents are significantly different.

3. Ignoring Confidence Scores

AI Builder provides a "confidence score" for every extraction. If you are building an automated system, always check this score. If the confidence is low (e.g., below 70%), route the document to a human-in-the-loop review process rather than automatically injecting the data into your database.

Callout: The Human-in-the-Loop Pattern Never assume 100% accuracy. Always design your workflows to handle low-confidence scenarios. A common industry pattern is:

1. AI extracts data.
2. If confidence score > 90%, proceed to automation.
3. If confidence score < 90%, send an email to a human operator with a link to the document for manual verification.

Comparison: AI Builder vs. Traditional OCR

It is helpful to understand how AI Builder compares to traditional Optical Character Recognition (OCR) tools. Traditional OCR is a "dumb" technology; it converts pixels to text but has no concept of what that text means.

As shown in the table, traditional OCR is sufficient if you only need the raw text from a document. However, if you need to extract specific business data—like the "Total" from an invoice—AI Builder is significantly more efficient because it handles the logic of finding and structuring that data for you.

Advanced Configuration: Using "Key-Value" Pairs

Sometimes, you may have documents where the label and the value are not always in the same place. AI Builder handles this by identifying the relationship between the label (e.g., "Invoice Date:") and the value (e.g., "12/01/2023"). When you are labeling your documents, make sure you are capturing both the label and the value area if necessary. This helps the model understand that the value is dependent on the presence of the label in its vicinity.

Security and Governance

As an administrator, you must ensure that your AI models are secure. AI Builder models are stored as part of the environment, meaning they adhere to the same Dataverse security roles as other components. Ensure that only authorized users have the permissions to create, edit, or publish models.

Furthermore, consider the sensitivity of the data being processed. If you are processing documents that contain Personally Identifiable Information (PII) or Protected Health Information (PHI), ensure that your environment is configured to meet your organization's compliance standards. AI Builder processes data within the region where your environment is hosted, which is a key consideration for data residency requirements.

Troubleshooting Performance Issues

If your model is not performing as expected, perform a systematic audit:

1. Check for "Overfitting": If the model performs perfectly on the training data but fails on new documents, you may have overfit the model. Try adding more diverse training documents to broaden the model's understanding.
2. Review Image Quality: Are your training documents too dark or skewed? Use pre-processing steps (like deskewing or contrast adjustment) if your source documents are low quality.
3. Check Field Alignment: Are your bounding boxes too tight? Ensure you include a little bit of whitespace around the text to help the model identify the field boundaries more clearly.

Key Takeaways

Configuring Document Processing models is a powerful way to eliminate manual data entry in your organization. By following the best practices outlined in this lesson, you can build reliable, scalable AI solutions.

• Start with Quality Data: The performance of your AI model is entirely dependent on the quality and diversity of your training documents.
• Use Collections: Organize your documents into collections to help the model understand different layouts and templates within the same model.
• Implement Human-in-the-Loop: Always account for scenarios where the AI’s confidence is low by routing those documents to a manual review queue.
• Iterate and Retrain: View your models as living assets. As your business documents change, your model must be updated and retrained to maintain accuracy.
• Leverage Power Automate: The real power of Document Processing is in the automation of downstream tasks. Use the AI Builder connector to bridge the gap between document ingestion and data storage.
• Monitor Performance: Regularly review the accuracy metrics provided in the AI Builder dashboard to proactively identify when a model needs adjustment.
• Security First: Always ensure that your AI models are managed within a secure environment, especially when handling sensitive or regulated data.

By adhering to these principles, you will be well-equipped to manage, configure, and maintain Document Processing models that drive real business value across your environment. Remember that AI is a tool to augment your processes, and the most successful implementations are those that combine the speed of automation with the oversight of human expertise.