Implementing Change Feed Notifications

Implementing Change Feed Notifications in Azure Cosmos DB

Introduction: The Power of Event-Driven Architectures

In the world of modern cloud applications, data is rarely static. When a user updates their profile, a sensor sends a temperature reading, or an e-commerce order is placed, that data needs to trigger a cascade of downstream actions. Traditionally, developers might have relied on polling—repeatedly asking a database "has anything changed?"—but this is inefficient, resource-heavy, and introduces unnecessary latency. Enter the Azure Cosmos DB Change Feed.

The Change Feed is a persistent log of all changes made to your Cosmos DB container, ordered by the time they were modified. It acts as a reliable, asynchronous stream of events that your applications can subscribe to. By using the Change Feed, you move from a reactive model where you constantly check for data updates to a proactive, event-driven model where your system reacts the moment a document is created, updated, or deleted.

Why does this matter? Because it decouples your services. Your primary application can focus on writing data to the database, while background processes (like indexers, analytics engines, or notification services) consume the Change Feed to perform their specific tasks. This architecture is the backbone of highly scalable systems that need to maintain synchronization across multiple microservices or storage systems without slowing down the primary data write path.

Understanding the Mechanics of the Change Feed

At its core, the Change Feed is a sorted list of documents in the order they were modified. If a document is updated multiple times, only the latest version of the document is guaranteed to appear in the feed for that specific change. It is important to note that the Change Feed does not store the history of every intermediate state; it provides a view of the current state of a document at the time of modification.

The Change Feed works by tracking the "continuation token" of the stream. This token is essentially a bookmark that tells the system where the last consumer stopped reading. When you start a process to read the Change Feed, you provide this token, and Cosmos DB serves up the changes that occurred since that specific point in time. If no token is provided, the feed can either start from the beginning of time (all existing documents) or from the current moment (only new changes).

Key Characteristics of the Change Feed

• Ordered: Changes are delivered in the order they were modified.
• Persistent: The feed is stored in the database, meaning it doesn't expire unless you define a time-to-live (TTL) policy that clears the actual data.
• Asynchronous: It does not block the primary write operations.
• At-least-once delivery: While rare, it is possible for a change to be delivered more than once, so your processing logic should be idempotent.

Callout: Change Feed vs. Polling Traditional polling requires a client to query the database repeatedly for updates, consuming Request Units (RUs) and creating unnecessary load on the database engine. The Change Feed, conversely, pushes data to the consumer. This reduces the total cost of operation and ensures that your application responds to data changes in near real-time, rather than waiting for the next polling interval to trigger.

Setting Up the Change Feed Processor Library

While you can interact with the Change Feed directly through the Cosmos DB SDKs, the recommended approach for most production scenarios is to use the Change Feed Processor (CFP) library. The CFP simplifies the complexity of distributing the workload across multiple instances and managing the state of the feed.

The processor manages the "lease" mechanism. If you have multiple instances of your application running, the CFP automatically partitions the work among them. If one instance crashes, the others detect the failure and take over the partitions that were being processed by the dead instance. This ensures high availability and resilience without you having to write complex coordination logic.

Prerequisites for Implementation

Before writing code, you need to set up two things in your Cosmos DB account:

1. The Monitored Container: The container where your primary data resides.
2. The Lease Container: A separate container used by the CFP to store the state (the bookmarks or continuation tokens) of the processing.

Tip: Lease Container Sizing The lease container is a small, lightweight storage area. You can usually configure it to have a very low Request Unit (RU) throughput—often just 400 RUs is sufficient—because it only performs small read/write operations to track the progress of your processor.

Step-by-Step: Implementing a Change Feed Processor

To implement a Change Feed Processor in a .NET application, follow these steps.

Step 1: Install the Required NuGet Package

You need the Microsoft.Azure.Cosmos library. You can add it via your IDE's package manager or by running the following command in your terminal:

dotnet add package Microsoft.Azure.Cosmos

Step 2: Configure the Processor

You must define the processor, point it to the monitored container, define a delegate for handling changes, and provide a connection to the lease container.

using Microsoft.Azure.Cosmos;

// Setup the Cosmos Client
CosmosClient client = new CosmosClient("your-connection-string");

Container monitoredContainer = client.GetContainer("database", "orders");
Container leaseContainer = client.GetContainer("database", "leases");

// Build the processor
ChangeFeedProcessor processor = monitoredContainer.GetChangeFeedProcessorBuilder<Order>(
        processorName: "OrderProcessingProcessor",
        onChangesDelegate: HandleChangesAsync)
    .WithInstanceName("instance-1")
    .WithLeaseContainer(leaseContainer)
    .Build();

Step 3: Define the Change Handler

The handler is the method that executes every time a batch of changes is received from the feed.

static async Task HandleChangesAsync(
    IReadOnlyCollection<Order> changes,
    CancellationToken cancellationToken)
{
    foreach (Order order in changes)
    {
        Console.WriteLine($"Processing order: {order.Id}");
        // Perform downstream tasks here, like sending an email or updating an index
    }
}

Step 4: Start and Stop the Processor

The processor is a long-running service. You start it when your application initializes and stop it when the application shuts down.

await processor.StartAsync();

// ... application runs ...

await processor.StopAsync();

Advanced Scenarios: Handling Deletions and Large Data

One common question is: "How do I handle deletions?" By default, the Change Feed shows inserts and updates. If you need to track deletions, you must enable "Soft Deletes" in your application logic. Instead of actually deleting a document, you update a flag (e.g., isDeleted: true). Your Change Feed processor then detects this update, performs the necessary cleanup logic, and then you can use a TTL policy on the container to eventually remove the "soft-deleted" document from the database automatically.

Dealing with Large Documents

If your documents are very large, the Change Feed might return fewer documents per batch to avoid exceeding the maximum message size. The Change Feed Processor handles this transparently by grouping changes into batches. However, you should ensure that your HandleChangesAsync method is efficient. If you perform long-running operations inside this method, you will block the processor from moving to the next batch.

Warning: Blocking the Processor Never perform long-running blocking operations inside the onChangesDelegate. If the handler takes too long, the processor will be delayed in updating the lease, which can lead to lag in your event processing. If you have heavy work to do, offload it to a background queue (like Azure Service Bus or RabbitMQ) and have the Change Feed Processor simply act as a producer to that queue.

Best Practices for Production

When scaling Change Feed implementations, consider these industry standards to maintain reliability and performance:

1. Idempotency is Non-Negotiable: Because the Change Feed can, in rare circumstances, deliver the same event twice, your downstream logic must be idempotent. If you are inserting a record into a SQL database based on a Cosmos update, check if the record exists before inserting or use an "upsert" command.
2. Monitor Your Lag: You should monitor the "Change Feed Lag," which is the difference between the latest change in the container and the last change processed by your consumer. Azure Monitor provides metrics for this. If the lag increases, it means your processors cannot keep up with the rate of changes.
3. Use Multiple Instances: To handle high volumes of changes, deploy multiple instances of your processor. The CFP will automatically balance the partitions across these instances. Ensure each instance has a unique InstanceName.
4. Error Handling: Always wrap your logic inside the onChangesDelegate in a try-catch block. If your code throws an unhandled exception, the processor will stop. You need to decide whether to log the error and skip the problematic document, or halt the processor to prevent data inconsistency.

Comparison Table: Change Feed vs. Alternatives

Integrating with Azure Functions

Azure Functions provides a built-in "Cosmos DB Trigger" that abstracts almost all of the boilerplate code mentioned above. This is often the preferred way to implement Change Feed logic because it scales automatically and manages the lease container for you.

When using an Azure Function, you simply define the trigger in your function signature:

[FunctionName("ProcessOrderChanges")]
public static void Run(
    [CosmosDBTrigger(
        databaseName: "orders-db",
        containerName: "orders",
        Connection = "CosmosConnectionString",
        LeaseContainerName = "leases")] IReadOnlyList<Document> documents,
    ILogger log)
{
    foreach (var doc in documents)
    {
        log.LogInformation($"Document modified: {doc.Id}");
    }
}

This approach is extremely efficient for serverless architectures. You don't have to manage the lifetime of the ChangeFeedProcessor object; the Azure Functions runtime handles the starting, stopping, and scaling of your code based on the volume of changes.

Common Pitfalls and How to Avoid Them

1. The "Poison Document" Problem

A "poison document" is a document that causes your processing logic to crash. If your handler tries to process a document, hits an exception, and crashes, the processor will retry. If the retry also fails, the processor may enter a loop, repeatedly attempting to process the same faulty document.

• Solution: Implement a dead-letter queue or a logging mechanism. If an exception occurs, log the document ID and the error details to a separate table or storage, and then continue with the next document. Do not let one bad document block the entire pipeline.

2. Over-Provisioning Throughput

Developers often over-provision the RUs for the monitored container because they are worried about the Change Feed slowing down the main application.

• Solution: Remember that the Change Feed is a separate read path. While it does consume RUs, it is optimized. Monitor your RU usage and start with a conservative number. Only scale up if you see 429 (Too Many Requests) errors.

3. Ignoring Partition Key Design

The Change Feed is partition-aware. If you have a poorly designed partition key (e.g., a "hot partition" where one partition gets 90% of the traffic), the Change Feed processor will also struggle with that specific partition.

• Solution: Ensure your partition key has high cardinality. This distributes the data—and the subsequent Change Feed processing workload—evenly across the underlying physical partitions of the container.

Callout: The Importance of Idempotency Idempotency means that performing the same operation multiple times results in the same outcome as performing it once. In Change Feed processing, if your service crashes after updating a downstream system but before updating the lease, the system will restart and try to process that same change again. If your logic is not idempotent, you might double-charge a customer or duplicate a log entry. Always design your downstream logic to be "safe" to run multiple times for the same input.

Advanced Topics: Filtering and Projection

Sometimes, you don't need every change. Perhaps you only care about "Order Shipped" events. In the past, you would have to process every change and filter it inside your code. However, with newer versions of the SDK and integration with Azure Functions, you can sometimes apply server-side filtering or use specific change feed modes.

While the standard Change Feed provides the full document, you can also use the "Latest Version" mode. If your application only cares about the current state and doesn't need to see every single update that happened within a millisecond, the Change Feed processor can be configured to focus on the most recent state, effectively ignoring transient intermediate states.

Practical Example: Downstream Synchronization

A classic use case is keeping a Search Index (like Azure Cognitive Search) in sync with your Cosmos DB database.

1. Write: The application writes a product update to Cosmos DB.
2. Trigger: The Change Feed processor detects the change.
3. Process: The processor extracts the relevant fields (Name, Price, Category).
4. Sync: The processor pushes these fields to the Search Index via an API call.
5. Result: The search index is updated within seconds of the database change without the main application ever knowing about the search engine.

This pattern is highly effective for building "read-optimized" views. Your Cosmos DB is the "source of truth," and the Change Feed allows you to project that data into any format or storage medium required for your specific query needs.

Troubleshooting Performance

If you find that your processor is falling behind, you should check the following:

• Network Latency: Is your processor running in the same Azure region as your Cosmos DB account? Cross-region processing will significantly increase latency and decrease throughput.
• RU Limitations: Check the RequestCharge property in the response headers. If it is consistently hitting your limit, you may need to increase the RU/s of the monitored container.
• Batch Size: You can configure the MaxItems parameter in your processor. If you have a very high volume of small changes, increasing the batch size can improve throughput by reducing the number of round-trips to the database.
• CPU/Memory constraints: If your application is running in a container or a small virtual machine, ensure that you aren't CPU-bound. Processing changes requires parsing JSON and executing your business logic; if the host machine is at 100% CPU, the processor will stall.

Key Takeaways

Implementing Change Feed Notifications is a foundational skill for any developer working with Azure Cosmos DB. By shifting from polling to event-driven processing, you create applications that are more efficient, responsive, and easier to maintain.

1. Event-Driven Design: Use the Change Feed to decouple your services. Your database should be the primary record, and downstream systems (analytics, search, notifications) should react to changes via the feed.
2. Use the Processor Library: Don't reinvent the wheel. The Change Feed Processor (CFP) library handles partitioning, load balancing, and failure recovery automatically.
3. Idempotency is Key: Always design your downstream handlers to be idempotent. Assume the Change Feed might deliver the same event more than once.
4. Lease Management: The lease container is the "brain" of the processor. Keep it small, efficient, and well-maintained.
5. Monitor Your Lag: Proactive monitoring of your Change Feed lag is the best way to ensure your system remains responsive under load.
6. Avoid Blocking: Keep your change handler logic fast. If you have heavy processing to do, use an asynchronous queue to buffer the work.
7. Serverless Integration: When possible, use Azure Functions with the Cosmos DB Trigger. It provides the most simplified development experience while maintaining the power and scalability of the underlying Change Feed infrastructure.

By following these practices, you can build systems that handle massive amounts of data updates with grace and reliability, ensuring that your application remains performant even as your data volume grows. Remember, the Change Feed is not just a feature—it is a powerful tool to transform how your data flows through your entire cloud architecture.