Website and Document Sources

Module: Plan and Configure Agent Solutions

Lesson: Configuring Knowledge Sources – Websites and Documents

Introduction: The Foundation of Intelligent Agents

In the landscape of modern artificial intelligence, an agent is only as capable as the information it can access. Whether you are building a customer support bot, an internal research assistant, or a technical documentation guide, the "knowledge source" acts as the brain's long-term memory. Without curated, accurate, and structured data, an agent is forced to rely on its base training, which is often too generic to solve specific business problems or provide context-aware answers.

Knowledge sources generally fall into two primary categories: live web content and static documents. Configuring these sources requires more than just pointing an agent to a URL or a folder of PDFs. It requires a deep understanding of how data is ingested, cleaned, indexed, and retrieved. If you fail to manage these sources effectively, your agent will suffer from "hallucinations," outdated responses, or an inability to find the right information when it matters most. This lesson explores the technical and strategic nuances of integrating websites and documents into your agent solutions.

Understanding Knowledge Source Architecture

Before diving into the configuration, we must understand the pipeline. When an agent retrieves information, it follows a process known as Retrieval-Augmented Generation (RAG). The data must first be converted from a human-readable format (like an HTML page or a PDF document) into a machine-readable format (vector embeddings).

The quality of your agent’s output is directly tied to the quality of this ingestion process. If your web crawler pulls in navigation menus, footers, and advertisements, the agent will get "noisy" data that distracts from the core content. Similarly, if your documents contain broken tables or messy OCR (Optical Character Recognition) results, the agent may struggle to extract accurate insights.

Callout: Retrieval-Augmented Generation (RAG) Explained RAG is a framework that allows an AI model to look up information from an external source before generating an answer. Unlike a standard model that only uses its internal training data, a RAG-enabled agent performs a search query against your provided knowledge sources, retrieves the most relevant chunks of text, and uses that text to construct a precise, fact-based response.

Part 1: Configuring Website Sources

Websites are dynamic, living sources of information. They are ideal for knowledge bases, FAQs, and product documentation. However, configuring them requires careful planning regarding scope and frequency.

Defining the Crawl Scope

When you provide a URL, the agent needs to know how deep it should go. A common mistake is allowing an agent to crawl an entire domain, including login pages, shopping carts, or irrelevant social media links. You must define a "crawl boundary" to ensure the agent stays within the relevant documentation.

• Include Patterns: Define specific sub-paths that the agent is allowed to index (e.g., /docs/v2/guides/*).
• Exclude Patterns: Explicitly block paths that contain noise or private data (e.g., /login, /cart, /admin, or /search).

Handling Dynamic Content

Many modern websites use JavaScript frameworks like React or Vue. If you use a simple crawler, it might only see the initial HTML shell and miss the actual content rendered by the browser. When configuring your agent, ensure your ingestion engine supports headless browser rendering, which executes the JavaScript before scraping the page content.

Practical Example: Configuring a Web Scraper

If you are using a Python-based ingestion script, you might use a library like BeautifulSoup combined with Playwright. Here is a basic implementation pattern:

# Example: Basic structure for a web content fetcher
from playwright.sync_api import sync_playwright

def get_page_content(url):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto(url)
        # Wait for the main content to load
        page.wait_for_selector(".main-content")
        content = page.inner_text(".main-content")
        browser.close()
        return content

# This function extracts only the text within the 'main-content' div,
# ignoring navigation bars and footers that often confuse AI models.

Warning: Respecting Robots.txt Always verify that your agent respects the robots.txt file of the target website. Many sites explicitly forbid automated scraping. Ignoring these directives can lead to your IP being blacklisted or legal complications regarding data usage policies.

Part 2: Configuring Document Sources

Documents—such as PDFs, Word files, and text files—are often more "dense" than web pages. They contain specific organizational knowledge that isn't always available on the public web.

The Challenge of PDF Structure

PDFs are notoriously difficult for AI. They are designed for visual layout, not data structure. A PDF might look perfect to a human eye, but the underlying text stream might be out of order, or tables might be flattened into unreadable strings.

To successfully use PDFs as knowledge sources, you must perform "pre-processing":

1. Layout Analysis: Identify headers, footers, and body text to ensure they are parsed in the correct order.
2. Table Extraction: Use specialized tools (like Tabula or Amazon Textract) to convert visual tables into Markdown or CSV format before feeding them to the agent.
3. OCR for Scanned Docs: If your documents are images of text, you must run an OCR pass to generate searchable text layers.

Choosing the Right Chunking Strategy

Once you have the text, you cannot simply dump an entire 100-page manual into the agent’s context window. You must "chunk" the document. Chunking is the process of breaking long text into smaller, meaningful segments.

• Fixed-size chunking: Splitting by character count. This is easy but often breaks sentences or paragraphs in the middle.
• Semantic chunking: Splitting by logical sections, such as paragraphs or headers. This is the industry standard for high-quality retrieval.

Callout: The Importance of Metadata When storing your documents, always attach metadata such as "Source Title," "Last Updated Date," and "Department." This allows your agent to perform "filtered retrieval." For example, if a user asks about "Company Policy," the agent can filter the search to only look at documents with the "HR" tag, preventing it from accidentally pulling information from an outdated "Engineering" policy document.

Part 3: Maintaining Data Integrity and Freshness

A knowledge source is not a "set it and forget it" component. Information changes, and your agent must evolve with it. If your documentation is updated but your agent is still referencing the old version, you have created a liability.

Implementing a Refresh Cycle

You need a strategy for keeping your knowledge sources up-to-date. There are three common approaches:

1. Scheduled Polling: The agent checks the source for changes every 24 hours. This is suitable for stable documentation.
2. Event-Driven Updates: The agent is triggered to re-index a document whenever it detects a change in the source (e.g., a webhook from your Content Management System).
3. Manual Refresh: A human administrator triggers an update after a major release or documentation overhaul.

Cleaning Your Data

Garbage in, garbage out is the cardinal rule of AI. Before your data is indexed, run it through a cleaning pipeline:

• Remove boilerplate text (e.g., "Copyright 2023," "Click here to unsubscribe").
• Normalize formatting (e.g., convert all headers to a standard Markdown format).
• Remove sensitive information (PII) using regex patterns or dedicated privacy tools.

Comparison of Knowledge Source Types

Best Practices for Knowledge Configuration

1. Start Small: Do not try to ingest your entire corporate history at once. Start with a high-quality subset of documents or a single section of your website. Measure the accuracy of the agent, and expand only when you are satisfied with the performance.
2. Prioritize Markdown: Convert all your documents to Markdown before indexing. Markdown is the "native language" of most LLMs; it preserves hierarchy (headers, lists) and is much cleaner than raw PDF or Word text.
3. Use Human-in-the-Loop (HITL) Validation: Set up a testing environment where you can ask the agent questions and verify if the answers are being pulled from the correct source. If the agent cites the wrong document, you know you need to adjust your retrieval logic.
4. Monitor Retrieval Metrics: Track "Retrieval Precision" and "Retrieval Recall." Precision measures how many of the retrieved chunks were actually relevant. Recall measures if the agent found all the relevant information available for a specific query.

Note: When configuring your agent, always provide it with a "system prompt" that instructs it on how to handle missing information. For example: "If the answer is not found in the provided knowledge sources, state that you do not have that information, rather than attempting to guess." This significantly reduces hallucination rates.

Common Pitfalls and How to Avoid Them

Pitfall 1: Over-indexing

Many developers index every single page they can find. This leads to "retrieval noise," where the agent gets confused by similar but slightly different documents.

• The Fix: Use a "whitelist" approach. Only index the specific pages or documents that contain the information the agent actually needs to perform its job.

Pitfall 2: Ignoring Document Context

If you have a document that is 50 pages long, splitting it into random 500-character chunks will destroy the context. The agent will have no idea what the document is about because the title and header are separated from the text.

• The Fix: Implement "context-aware chunking." Every chunk should be appended with the document title and a summary of the section it belongs to.

Pitfall 3: Stale Data

If your website updates daily but your agent only re-indexes weekly, your agent will provide incorrect information for six days out of the week.

• The Fix: Use a versioning system. Include a "last updated" timestamp in your document metadata. If the agent retrieves a document that is older than a certain threshold, it can be programmed to warn the user that the information might be outdated.

Step-by-Step: Setting Up a Knowledge Source

Follow this workflow to configure a new document source for your agent:

1. Preparation: Collect all relevant source documents in a centralized folder (e.g., a secure S3 bucket or a local repository).
2. Conversion: Convert all files into a unified format, preferably Markdown or plain text. Remove all images, tables, and complex formatting that might cause parsing errors.
3. Metadata Injection: Add a header to each file containing metadata, such as the document’s purpose, the team responsible for it, and the date of creation.
4. Ingestion: Use your agent’s ingestion API to upload these files. Ensure you are using a consistent chunking strategy (e.g., 500-character chunks with a 50-character overlap to preserve context at boundaries).
5. Testing: Perform a series of "known-answer" tests. Ask questions that you know are contained within the documents and verify that the agent cites the correct source.
6. Refinement: If the agent fails to find the correct information, check your retrieval settings. You may need to increase the number of chunks retrieved or adjust the similarity threshold in your vector database.

Quick Reference: Configuration Checklist

• Scope Defined: Have you limited the crawler to the correct domain/folder?
• Format Cleaned: Are your documents free of noise, headers, and footers?
• Chunking Strategy: Is your chunk size appropriate for the complexity of the content?
• Metadata Added: Does each chunk contain context about the source?
• Refresh Policy: Is there an automated or manual plan for updating the data?
• Privacy Check: Have you redacted PII from the documents?

Frequently Asked Questions (FAQ)

Q: Can I use both websites and documents at the same time? A: Yes, most modern agent frameworks allow you to combine multiple knowledge sources. The agent will perform a search across all indexed sources simultaneously.

Q: How do I prevent the agent from using outdated information? A: You can implement a "TTL" (Time to Live) on your data. If the document hasn't been re-indexed within a set timeframe, the agent can be instructed to ignore it or flag it as potentially stale.

Q: What is the ideal chunk size? A: There is no single "ideal" size. However, 300 to 500 tokens is a good starting point for most general-purpose documentation. If your documents are highly technical, you might want smaller chunks to maintain high precision.

Q: My agent keeps hallucinating even though the document is in the knowledge base. Why? A: This usually happens because the retrieval step failed to find the specific chunk. Try improving your query expansion (using the agent to rewrite the user's question before searching) or using a more robust search algorithm like "Hybrid Search" (which combines keyword search with vector search).

Key Takeaways

1. Data Quality is Everything: Your agent’s intelligence is directly limited by the quality of the data you provide. Invest time in cleaning and formatting your sources before ingestion.
2. Structure Matters: Use Markdown for your documents to ensure that the agent understands the hierarchy, headers, and lists within your content.
3. Chunking is a Strategic Choice: Do not use arbitrary chunk sizes. Use context-aware strategies that keep related information together and include metadata to help the agent understand what it is reading.
4. Manage the Scope: Avoid the temptation to index everything. Keep your knowledge sources focused on the specific domain the agent is intended to cover to prevent noise and retrieval errors.
5. Keep it Fresh: Treat your knowledge sources as living systems. Establish a clear refresh cycle to ensure that users are not receiving outdated or incorrect information.
6. Test and Iterate: Retrieval is rarely perfect on the first try. Use a test set of questions to evaluate your retrieval precision and refine your configurations accordingly.
7. Respect Privacy and Compliance: Always scrub documents for personal information and ensure that your web crawling activities comply with the site's terms of service and robots.txt files.

By mastering these configuration strategies, you move from simply "giving an agent a file" to architecting a robust, reliable knowledge system. This foundation allows your agents to act as genuine experts in your organization's specific domain, providing accurate, timely, and context-rich assistance to your users. Remember that the goal is not just to provide information, but to ensure that the right information is retrieved at the right time.