Topics and Subscriptions

Module: Connect and Consume Azure Services

Section: Event and Message-Based Solutions

Lesson: Topics and Subscriptions (Azure Service Bus)

Introduction: Why Decoupled Communication Matters

In the modern landscape of cloud architecture, building systems that are tightly coupled is a recipe for maintenance nightmares. When Service A must wait for Service B to respond directly, a failure in Service B causes a ripple effect, often crashing Service A or causing data loss. To build reliable, scalable distributed systems, we need a way to communicate that doesn't require both sides to be online and functional at the exact same moment. This is where messaging patterns, specifically the Publish/Subscribe (Pub/Sub) model, become essential.

Azure Service Bus Topics and Subscriptions offer a powerful implementation of this pattern. Unlike a simple queue, where one sender puts a message in and one receiver takes it out, a Topic acts as a broker that receives a message and then distributes it to multiple, independent subscriptions. Each subscription can have its own filtering rules, allowing different downstream services to receive only the information they care about. This architecture allows you to scale your services independently, introduce new functionality without breaking existing producers, and build resilient workflows that handle traffic spikes without losing data.

Understanding the Core Architecture

At its heart, an Azure Service Bus Topic is a specialized entity that acts as an "inbox" for messages sent by publishers. When a message arrives at the topic, it doesn't stay there; it is evaluated against the rules defined for each subscription attached to that topic. Think of a topic as a newsstand and the subscriptions as individual mailboxes for different newspapers. When a publisher sends a message, it is effectively "published" to the newsstand, and the service bus then delivers a copy of that message to every mailbox (subscription) that meets the criteria.

Key Components of the Pub/Sub Model

• The Publisher: The application or service that sends the message to the Topic. It does not need to know who is listening or how many subscribers exist. It simply hands off the data to the Service Bus.
• The Topic: The central communication hub. It acts as the buffer and the logic controller that determines how messages are routed.
• The Subscription: A virtual entity that receives a copy of the message from the Topic. Each subscription maintains its own message queue, meaning if one subscriber is slow, it doesn't impact the performance of other subscribers.
• The Consumer (Subscriber): The application that pulls messages from its specific subscription. It operates on its own schedule, processing messages as it is able.

Callout: Topics vs. Queues While queues are ideal for point-to-point communication where one message is processed by exactly one consumer, Topics and Subscriptions are designed for one-to-many communication. If you have a scenario where an order placement needs to trigger an email notification, update inventory, and log data for analytics, a Topic is the correct choice. Using a queue for this would require complex manual forwarding, whereas a Topic handles the distribution automatically.

Configuring Your First Topic and Subscription

Setting up an Azure Service Bus namespace is the first step toward implementing this pattern. A namespace acts as a container for all your messaging components. Once the namespace is created, you define Topics within it, and then define one or more Subscriptions for each Topic.

Step-by-Step Configuration via Azure Portal

1. Create a Service Bus Namespace: Navigate to the Azure portal, search for "Service Bus," and create a new resource. Choose a pricing tier—Standard or Premium are required for Topics and Subscriptions.
2. Create a Topic: Inside your namespace, select "Topics" from the left-hand menu and click "+ Topic." Give it a clear name that reflects the type of messages it will handle (e.g., order-events).
3. Define Subscriptions: Once the topic is created, click on it and select "+ Subscription." Give the subscription a name (e.g., inventory-service-sub).
4. Set Time to Live (TTL): You can define how long a message stays in the subscription before it expires. This is vital for time-sensitive data.
5. Enable Dead-Lettering: Ensure that "Dead-lettering on message expiration" is enabled. This ensures that if a message cannot be processed, it isn't simply deleted; it is moved to a special sub-queue for investigation.

Message Filtering: The Power of Targeted Delivery

One of the most useful features of Service Bus Subscriptions is the ability to filter incoming messages. You might have a topic that receives every order, but your "International Shipping Service" only wants to see orders where the country is not the United States. Instead of your service receiving every single message and discarding the ones it doesn't need, you can offload this logic to the Service Bus itself.

Types of Filters

• SQL Filters: These use a SQL-like syntax to evaluate message properties. For example, Country = 'UK' or OrderValue > 100.
• Boolean Filters: Simple "True" or "False" filters. True means the subscription receives all messages; False means it receives none.
• Correlation Filters: These are highly efficient filters that match against specific headers in the message, such as CorrelationId or Label. They are faster than SQL filters because they use exact matching rather than evaluation.

Note: Filters are evaluated at the time the message arrives at the topic. Changing a filter does not retroactively change what is already in the subscription queue; it only affects new, incoming messages.

Implementation: Writing the Code

To interact with Service Bus, we use the Azure.Messaging.ServiceBus NuGet package. This library provides a clean, asynchronous API for both sending and receiving messages.

Sending a Message (The Publisher)

The publisher is straightforward. You create a ServiceBusClient, create a ServiceBusSender for your specific topic, and send a ServiceBusMessage.

using Azure.Messaging.ServiceBus;

string connectionString = "<your-connection-string>";
string topicName = "order-events";

// Create the client
await using var client = new ServiceBusClient(connectionString);
ServiceBusSender sender = client.CreateSender(topicName);

// Create the message
string messageBody = "Order ID: 12345, Country: UK";
ServiceBusMessage message = new ServiceBusMessage(messageBody);

// Add custom properties for filtering
message.ApplicationProperties.Add("Country", "UK");

// Send the message
await sender.SendMessageAsync(message);

Receiving a Message (The Subscriber)

The receiver creates a ServiceBusProcessor. This processor acts like a background listener that triggers a callback function whenever a new message arrives in the subscription.

using Azure.Messaging.ServiceBus;

string connectionString = "<your-connection-string>";
string topicName = "order-events";
string subscriptionName = "inventory-service-sub";

await using var client = new ServiceBusClient(connectionString);
ServiceBusProcessor processor = client.CreateProcessor(topicName, subscriptionName);

// Define the handler
processor.ProcessMessageAsync += async args =>
{
    string body = args.Message.Body.ToString();
    Console.WriteLine($"Received: {body}");
    
    // Complete the message (removes it from the queue)
    await args.CompleteMessageAsync(args.Message);
};

// Define the error handler
processor.ProcessErrorAsync += args =>
{
    Console.WriteLine(args.Exception.ToString());
    return Task.CompletedTask;
};

// Start processing
await processor.StartProcessingAsync();

Best Practices for Production Systems

When moving from development to production, you must consider the reliability and performance characteristics of your messaging infrastructure. Service Bus is highly capable, but misconfiguration can lead to bottlenecks or data loss.

1. Always Use Asynchronous Patterns

Service Bus operations are I/O bound. Always use the await keyword with your operations to avoid blocking threads. In high-throughput systems, blocking threads can quickly lead to thread pool starvation and application crashes.

2. Implement Dead-Letter Queues (DLQ)

Never ignore the dead-letter queue. If a message fails processing repeatedly, it is moved to the DLQ. You should have a monitoring process or a separate "retry service" that periodically checks the DLQ to identify why messages are failing—is it a malformed message? A service dependency that is down? A logic error?

3. Manage Message Size

Standard Service Bus topics have a maximum message size of 256 KB. If you have large payloads, you should follow the "Claim Check" pattern. Upload the large data to Azure Blob Storage, and send a message containing only the URL/reference to that data in the Service Bus message.

4. Configure Auto-Delete on Idle

If you are creating dynamic subscriptions (e.g., for temporary user sessions), ensure you set the AutoDeleteOnIdle property. This prevents your namespace from becoming cluttered with thousands of orphaned subscriptions that no one is listening to.

5. Monitor Throughput and Throttling

Use Azure Monitor to track the "Active Messages" and "Incoming/Outgoing Requests" metrics. If you see high latency, it may be time to scale your namespace. Note that the "Basic" tier does not support topics; the "Standard" tier is the minimum, and "Premium" should be used for production workloads requiring high throughput or predictable latency.

Callout: The Claim Check Pattern When your message payload exceeds the 256 KB limit of a standard Service Bus message, do not try to compress it into the limit. Instead, store the large payload in an external store like Azure Blob Storage. Place a reference (the URI) to that blob in your message body. This keeps your messaging backbone fast and lean while allowing you to handle arbitrary data sizes.

Common Pitfalls and How to Avoid Them

Even experienced developers can run into issues with Service Bus. Below are the most common mistakes and how to steer clear of them.

• Forgetting to Complete Messages: In the code example above, args.CompleteMessageAsync is crucial. If you don't call this, the message remains in the subscription, and after the lock duration expires, it will be delivered again (duplicate processing). Always ensure your logic completes or abandons the message.
• Assuming Ordering: While Service Bus provides "First-In-First-Out" (FIFO) guarantees within a single queue or subscription, you cannot assume that all messages across all subscriptions will be processed in the exact order they were sent if you have multiple instances of a receiver processing messages concurrently. If order is critical, use "Sessions" in Service Bus.
• Ignoring the Lock Duration: By default, when a consumer receives a message, it is locked for a short period (usually 30 seconds). If your processing logic takes longer than the lock duration, the message will be "unlocked" and delivered to another consumer, leading to dual processing. If your logic is slow, increase the lock duration on the subscription configuration.
• Over-Filtering: While filtering is powerful, complex SQL filters on every single message can add latency. Keep your filters simple and use properties that are easy to evaluate.

Quick Reference Table: Service Bus Features

Designing for Resiliency: The "Retry" Strategy

In a cloud environment, transient errors are inevitable. A network blip or a temporary service restart can cause a message delivery to fail. You should never write a subscriber that simply gives up on the first error.

Implement an "Exponential Backoff" strategy within your message processor. If a message processing fails, wait for a few milliseconds, then a few seconds, then a few more. If it continues to fail after a set number of attempts (e.g., 5 retries), move the message to the DLQ. Most SDKs have built-in retry policies, but you should verify they are configured to match the stability of your downstream dependencies.

Scaling Your Architecture

As your system grows, you may find that a single subscriber cannot keep up with the volume of messages coming into the Topic. Because Service Bus Subscriptions are independent, you can scale them horizontally without any coordination.

You can spin up ten instances of your "Inventory Service," all listening to the same subscription. The Service Bus will automatically load-balance the messages across these ten instances. This is a "Competing Consumers" pattern. As long as your message processing is idempotent—meaning processing the same message twice doesn't cause harm—this is the most efficient way to scale your throughput.

Advanced Topic: Message Sessions

Message sessions are a specialized feature in Service Bus that allows you to group related messages. In a standard subscription, messages are handled independently. However, if you are processing an order workflow, you might have a "Create Order," "Update Order," and "Delete Order" message. You want all of these to be processed in order by the same instance of a consumer to maintain state.

By setting a SessionId on the message, you tell Service Bus that these messages belong together. When a consumer connects to a session, it gains an exclusive lock on that session. No other consumer can touch messages with that same SessionId until the first consumer finishes. This is an essential tool for high-concurrency systems that require strict sequencing.

Summary and Key Takeaways

Mastering Azure Service Bus Topics and Subscriptions is a fundamental skill for any cloud developer. It shifts your mindset from "how do I connect these two services" to "how do I build an event-driven ecosystem."

Key Takeaways:

1. Decoupling is Essential: Use Topics and Subscriptions to decouple your services, allowing them to scale independently and survive temporary outages.
2. Pub/Sub is Powerful: Move away from point-to-point queues when you have multiple consumers that need to process the same event data.
3. Filtering Saves Resources: Use SQL or Correlation filters to ensure your services only consume the data they actually need, reducing compute costs and processing latency.
4. Manage Your Locks: Always be mindful of your message lock duration. If your code takes longer than the lock, you will experience duplicate processing.
5. Never Ignore the DLQ: The Dead-Letter Queue is your most valuable tool for identifying bugs and transient failures. Monitor it closely.
6. Scale via Competing Consumers: Leverage the ability to have multiple instances of a subscriber competing for messages to handle high traffic volumes.
7. Choose the Right Tier: Use the Standard tier for development and low-to-medium traffic, and upgrade to Premium for production environments where performance and latency are critical requirements.

By applying these concepts, you will move from building fragile, monolithic interactions to building resilient, distributed systems that can handle the unpredictability of real-world cloud traffic. Start by implementing a simple publisher/subscriber pair, then incrementally add filtering and error handling as your application needs evolve.

FAQ: Frequently Asked Questions

Q: Can I change a subscription filter after it's been created? A: Yes, you can update the filter rule on a subscription at any time via the Azure Portal or the SDK. Note that this only affects messages that arrive after the change is applied.

Q: What happens if I don't have enough subscribers? A: If a message is sent to a topic and there are no matching subscriptions, the message is simply discarded by the Service Bus. Always ensure you have at least one subscription that will catch your important messages.

Q: Is Service Bus better than Event Grid? A: They serve different purposes. Event Grid is for "reactive" events (e.g., a file was uploaded to storage). Service Bus is for "durable" messaging (e.g., order processing). Use Service Bus when you need guaranteed delivery, sessions, and complex routing.

Q: How do I handle duplicate messages? A: While Service Bus guarantees "at least once" delivery, it does not guarantee "exactly once." You should implement idempotency in your consumer code—check if a message has already been processed (perhaps by checking a database record) before performing the action.

Q: Are there limits to how many subscriptions I can have per topic? A: There are limits based on your Service Bus tier. Check the official Azure documentation for the latest limits on entities per namespace. For most applications, the default limits are more than sufficient.