Azure Functions Change Feed Trigger

Module: Optimize Azure Cosmos DB Solution

Lesson: Azure Functions Change Feed Trigger

Introduction: Understanding the Change Feed

In the world of distributed databases, keeping secondary systems synchronized with your primary data store is a classic challenge. When you update a user profile, you might need to update a search index, send a welcome email, or aggregate metrics for a dashboard. Traditionally, this required polling the database repeatedly, which is inefficient, resource-heavy, and introduces significant latency. Azure Cosmos DB solves this with the Change Feed, a persistent log of all modifications made to items within your container.

The Change Feed acts as an immutable, append-only record of changes. When you pair this with Azure Functions, you create an event-driven architecture where your code executes automatically the moment a document is inserted or updated. This is not just a convenience; it is a fundamental shift in how you build modern, reactive applications. By mastering the Change Feed Trigger, you move away from scheduled batch jobs and toward real-time data processing, allowing your system to react instantly to user behavior or system events.

This lesson explores how to implement the Azure Functions Change Feed Trigger, the mechanics behind the scenes, and the best practices required to ensure your production workloads remain performant and cost-effective.

How the Change Feed Trigger Works

At its core, the Change Feed trigger is an abstraction over the ChangeFeedProcessor library. When you configure an Azure Function to listen to a Cosmos DB container, the runtime automatically manages the complexity of reading the feed, checkpointing progress, and scaling across multiple function instances.

When a document is created or modified in the source container, Cosmos DB writes the change to the internal log. The Azure Function polls this log. If it finds new changes, it packages them into a batch and passes them to your function code. You do not need to worry about managing the cursor or tracking which documents have already been processed; the Azure Functions runtime handles this state management using a separate "lease" container.

Callout: The Lease Container The lease container is the secret sauce that makes the Change Feed trigger reliable. It stores metadata about the current position (checkpoint) of the function in the feed. If your function crashes or needs to scale out, the new instances look at the lease container to determine exactly where the previous instance left off. Never delete or manually tamper with the lease container while your function is running, as this will cause the function to restart from the beginning of the feed.

Setting Up Your First Trigger

To get started, you need three primary components: a source Cosmos DB container, a lease container (which can be in the same database or a different one), and the Azure Function App itself.

Step-by-Step Configuration

1. Create the Source Container: Ensure your Cosmos DB container has a partition key defined. The Change Feed is ordered by the partition key, which is critical for performance.
2. Create the Lease Container: Create a separate container in your Cosmos DB account. This container is lightweight and only requires a partition key named /id.
3. Configure the Function: In your Azure Function project, add the Microsoft.Azure.WebJobs.Extensions.CosmosDB NuGet package.
4. Define the Trigger: Use the [CosmosDBTrigger] attribute in your function signature.

Here is a basic example of a C# Azure Function using the isolated worker model:

[Function("ProcessCosmosChanges")]
public void Run(
    [CosmosDBTrigger(
        databaseName: "InventoryDatabase",
        containerName: "Products",
        Connection = "CosmosDBConnection",
        LeaseContainerName = "leases",
        CreateLeaseContainerIfNotExists = true)] IReadOnlyList<Product> input,
    FunctionContext context)
{
    if (input != null && input.Count > 0)
    {
        foreach (var product in input)
        {
            // Logic to process the document
            Console.WriteLine($"Processing product: {product.Id}");
        }
    }
}

In this snippet, the input parameter is a IReadOnlyList<T>. It is important to note that the trigger often receives a batch of documents rather than a single document. This batching is an optimization to reduce the overhead of function invocations.

Best Practices for Production

Implementing a trigger is straightforward, but making it reliable in a high-throughput environment requires careful planning.

1. Idempotency is Mandatory

The Change Feed trigger guarantees "at-least-once" delivery. This means that under certain conditions—such as a function crash right before a checkpoint is saved—your function might receive the same document twice. Your code must be idempotent, meaning processing the same document multiple times should result in the same state. For example, if you are updating a SQL database based on a Cosmos DB change, use UPSERT logic rather than INSERT logic to prevent primary key constraint violations.

2. Keep Processing Logic Lightweight

Your function should perform the absolute minimum work necessary. If you need to perform long-running tasks, such as calling an external API or performing heavy data transformation, offload this work to a queue. Use the Change Feed trigger only to place the message onto an Azure Queue Storage or Service Bus queue, and have a separate function process the actual business logic. This keeps the Change Feed trigger fast and prevents it from falling behind the feed.

3. Monitor the "Lag"

One of the most important metrics for Change Feed performance is the "Change Feed Lag." This represents the time difference between when a change occurs in the source container and when it is processed by the function. If you notice your lag increasing, it means your function is not keeping up with the rate of changes. You may need to increase the number of function instances or optimize your processing code.

Warning: The Pitfall of "Poison" Documents If your function throws an unhandled exception while processing a batch, the trigger will retry the batch indefinitely. If the error is caused by a specific document in the batch (a "poison" document), the function will keep failing, effectively blocking the feed. Always wrap your processing logic in a try-catch block and implement custom logging to identify the problematic document so you can skip it or handle it gracefully.

Comparing Approaches: Polling vs. Change Feed

To appreciate the value of the Change Feed, it helps to compare it to the traditional polling method.

Handling Large Data Volumes

When your Cosmos DB container grows into the terabytes, the Change Feed remains performant because it is partitioned. The ChangeFeedProcessor automatically distributes the work across multiple function instances based on the partition keys.

If you have a massive amount of data being written, you can scale your Azure Function App's plan. If you are using a Consumption Plan, the platform will automatically scale the number of function instances to handle the incoming feed. However, if you are using a Premium or App Service Plan, you may need to configure autoscale rules based on CPU usage or memory to ensure you have enough compute power to keep up with the feed.

Managing Batch Size

You can control the number of documents processed in a single function invocation by adjusting the MaxItemsPerInvocation property in your trigger configuration. If your processing logic is computationally expensive, setting this to a lower value (e.g., 50 or 100) can prevent your function from hitting memory limits or execution time timeouts.

[CosmosDBTrigger(
    databaseName: "InventoryDatabase",
    containerName: "Products",
    Connection = "CosmosDBConnection",
    LeaseContainerName = "leases",
    MaxItemsPerInvocation = 100)]

Advanced Scenarios: Filtering and Handling Deletes

By default, the Change Feed includes all insert and update operations. Historically, it did not include deletes. However, with the introduction of "Change Feed with Soft Deletes" or using the "All Versions and Deletes" mode, you can now track deletions as well.

Implementing Soft Deletes

If you are not using the "All Versions and Deletes" mode, the most common pattern for tracking deletes is to implement a soft delete flag in your data schema. Instead of deleting a document, your application sets a property like isDeleted: true. Your Change Feed function then checks for this flag:

foreach (var item in input)
{
    if (item.IsDeleted)
    {
        // Logic to remove from search index or cache
        RemoveFromExternalSystem(item.Id);
    }
    else
    {
        // Logic to update or insert
        UpdateExternalSystem(item);
    }
}

This approach is highly recommended because it is cleaner, less error-prone, and works consistently across all versions of the Cosmos DB SDK.

Common Pitfalls and Troubleshooting

1. The Function is Not Triggering

If your function is not firing, check these common areas:

• Connection String: Ensure the CosmosDBConnection points to the correct account and database.
• Lease Container: Check if the lease container is being updated. If it is not, the function may not have permission to write to that container.
• Firewall Rules: If your Cosmos DB account is behind a virtual network or firewall, ensure your Azure Function App is configured with a Virtual Network integration to access the database.

2. High RU Consumption

If your Change Feed trigger is consuming too many Request Units (RUs), it is usually because the function is re-reading the same documents or the processing logic is too complex. Check your code for excessive queries back to the source container within the function. Remember, the document passed to the function already contains the data you need—do not perform a ReadItemAsync call unless you absolutely have to.

3. Execution Timeouts

If you are processing large batches, your function might exceed the maximum allowed execution time. This is common in Consumption Plans. If this happens, you have two options:

• Reduce MaxItemsPerInvocation.
• Optimize the code to be more efficient.
• Move the workload to a Durable Function, which can handle long-running processes more effectively by breaking them into smaller, stateful tasks.

Tip: Use Durable Functions for Complex Workflows If your Change Feed logic requires multiple steps, such as calling three different APIs and then updating a secondary database, do not try to squeeze that into a standard Azure Function. Use a Durable Function orchestration. The Change Feed trigger calls the orchestrator, which then manages the long-running process, providing built-in retries and state management.

Performance Tuning: The Hidden Configuration

While the standard settings work for most use cases, there are "hidden" configurations in the host.json file that can influence the behavior of the ChangeFeedProcessor.

• leaseAcquireInterval: Defines how often the function checks for new partitions to lease.
• leaseExpirationInterval: How long a function instance holds onto a lease before another instance can take over.
• feedPollDelay: How long the function waits between polling the Change Feed when no new changes are found.

You should rarely need to modify these, but if you are running a massive, high-concurrency system, tuning these values can help balance the sensitivity of your system to changes versus the overhead of polling.

Security Considerations

When connecting your Azure Function to Cosmos DB, avoid hardcoding connection strings in your code. Always use Application Settings or, better yet, Azure Key Vault. Furthermore, use Managed Identities to authenticate your Function App to the Cosmos DB account. This eliminates the need to manage secret keys entirely, as the identity of the Function App is used to grant access at the database level.

To configure a Managed Identity:

1. Enable System-Assigned Identity on your Function App.
2. Go to your Cosmos DB account in the Azure Portal.
3. Use the Access Control (IAM) blade to assign the "Cosmos DB Built-in Data Contributor" role to your Function App's identity.

This approach follows the principle of least privilege and significantly improves the security posture of your application.

Integration with Other Azure Services

The Change Feed trigger is often the starting point for a data pipeline. You can use it to:

• Push data to Azure Search: Sync your Cosmos DB data to an Azure Cognitive Search index for full-text search capabilities.
• Stream data to Power BI: Send document updates to an Event Hub, which can then be consumed by Stream Analytics to power real-time dashboards.
• Trigger Serverless Workflows: Use Logic Apps to send notifications or emails based on changes in your data.

By acting as the "glue" between your database and the rest of the Azure ecosystem, the Change Feed trigger allows you to build sophisticated architectures without needing to manage the underlying data synchronization logic yourself.

Summary and Key Takeaways

The Azure Functions Change Feed Trigger is a foundational tool for any developer working with Azure Cosmos DB. It transforms your database from a static storage bucket into a dynamic, event-driven engine.

To ensure your implementation is successful, keep these key takeaways in mind:

1. Prioritize Idempotency: Always write your processing logic so that it can handle the same document multiple times without causing side effects or duplicate data.
2. Batching is Your Friend: Understand that the trigger receives batches of items. Write your code to iterate over the collection and handle each item independently.
3. Keep the Trigger "Thin": Do not perform heavy lifting inside the trigger function. If the work is complex, offload it to a queue or a Durable Function to keep the feed moving.
4. Monitor Your Lag: Use Azure Monitor to track the Change Feed lag. If the lag is growing, your system is failing to process changes in real-time.
5. Use Managed Identities: Secure your connection by using Azure Managed Identities rather than connection strings to ensure your credentials are protected.
6. Handle Deletes Gracefully: Since standard Change Feed triggers don't always track deletes, use a "soft delete" pattern (a boolean flag) to notify your downstream systems when a document is removed.
7. Respect Partitioning: Ensure your logic is aware of the partition key, as the ChangeFeedProcessor distributes work based on partitions to achieve high performance.

By applying these principles, you will build applications that are not only performant and scalable but also resilient to the challenges of distributed systems. Whether you are building a simple microservice or a large-scale data processing pipeline, the Change Feed trigger is the most efficient way to react to your data.

Frequently Asked Questions (FAQ)

Q: Can I use one lease container for multiple functions? A: No. Each function trigger needs its own unique lease container. If you point two different functions to the same lease container, they will fight over the same partitions, leading to unpredictable behavior and data processing errors.

Q: What happens if I update a document 1,000 times in one second? A: The Change Feed only guarantees that you will see the latest version of the document for that specific change. If you update a document rapidly, the Change Feed might collapse those updates, ensuring that your downstream system eventually reaches the latest state, but it might not process every intermediate update.

Q: Can I change the partition key of my container after I start using the Change Feed? A: No. The partition key is immutable. Choose your partition key carefully at the start of your project, as it dictates how your data is distributed and how the Change Feed scales.

Q: Does the Change Feed affect the RU cost of my database? A: Yes, reading from the Change Feed consumes RUs. However, it is significantly more efficient than querying the container directly. The RUs consumed by the Change Feed are generally lower than the RUs required to perform the same read operations via standard point-reads or queries.

Q: Can I process the Change Feed from a different region? A: Yes, if you have multi-region replication enabled, you can read the Change Feed from any region where the data is replicated. This can be useful for geo-distributed applications that need to process data locally.

Q: Is there a limit to how many changes can be in a batch? A: You control the batch size with MaxItemsPerInvocation. There is no hard limit on the total number of changes in the feed, as it is a persistent log, but you should keep your batch sizes reasonable to ensure your function completes within the timeout limits of your hosting plan.