Rate Limiting and Throttling

Mastering Rate Limiting and Throttling in Azure API Management

Introduction: Protecting Your Infrastructure

In the modern landscape of distributed systems, your APIs are the front door to your business logic, data, and services. While you want this door to be open to legitimate users and applications, you must also defend it against accidental over-utilization and malicious intent. Rate limiting and throttling are the primary mechanisms used to control the flow of incoming traffic, ensuring that your backend services remain stable, performant, and available for everyone.

Rate limiting is the process of restricting the number of requests a user or client can make to your API within a specific timeframe. Throttling, while often used interchangeably, typically refers to the mechanism of slowing down or rejecting requests once a certain threshold is exceeded. Together, these practices form the backbone of API governance. Without them, a single rogue script or a sudden spike in traffic could overwhelm your backend resources, leading to cascading failures, increased latency, or complete service outages.

Azure API Management (APIM) provides a powerful suite of tools to implement these controls without requiring changes to your backend code. By configuring policies at the gateway level, you gain granular control over how your APIs are consumed. This lesson will guide you through the theory, implementation, and best practices for managing traffic flow, protecting your resources, and ensuring a fair distribution of service capacity.

Understanding the Core Concepts

Before we dive into the technical implementation, it is essential to distinguish between the various ways we control traffic. In Azure APIM, these controls are implemented via "policies," which are XML-based configurations that execute sequentially as requests flow through the gateway.

Rate Limiting vs. Throttling

• Rate Limiting: This is a proactive measure. You define a "budget" for a user or client (e.g., 100 requests per minute). Once the client hits that limit, the gateway rejects subsequent requests until the time window resets. This is primarily used to prevent abuse and enforce usage tiers.
• Throttling: This is often a reactive or protective measure. You might throttle traffic when your backend service reports that it is struggling. While APIM doesn't automatically "throttle" in the sense of adding latency, it uses the rate-limit-by-key and quota policies to effectively throttle incoming traffic to protect backend health.

Callout: The Difference Between Rate Limiting and Quotas A rate limit is time-bound and short-term, such as "10 requests per second." It is designed to prevent bursts of traffic from overwhelming the system. A quota, however, is long-term and cumulative, such as "10,000 requests per month." Quotas are typically used for business-level billing or tiering, whereas rate limits are used for operational stability.

Implementing Rate Limiting Policies in Azure APIM

Azure APIM offers two primary policies for traffic control: rate-limit and rate-limit-by-key. Understanding when to use each is critical for effective API management.

The rate-limit Policy

The rate-limit policy is the simplest form of traffic control. It applies a global limit to all calls made through the scope where the policy is defined. For example, if you apply this to an entire API, every single consumer of that API shares the same bucket of requests.

<rate-limit calls="100" renewal-period="60" />

In this example, the gateway allows 100 calls per 60 seconds. Once the 100th call is made, all subsequent calls within that 60-second window will receive a 429 Too Many Requests status code. This is useful for preventing global spikes, but it is rarely enough for production scenarios because it treats all users as a single entity.

The rate-limit-by-key Policy

The rate-limit-by-key policy is far more flexible. It allows you to define a "key" based on which the rate limit is tracked. This key can be the user's IP address, an API subscription key, a custom header, or even a claim from a JWT token.

<rate-limit-by-key calls="10" 
                   renewal-period="60" 
                   counter-key="@(context.Request.IpAddress)" />

By using context.Request.IpAddress, you ensure that each individual client is limited to 10 requests per minute. If one user starts flooding your API, they are blocked, but other users remain unaffected.

Note: Always consider the implications of using IP addresses as keys. In environments where users are behind a corporate proxy or a NAT (Network Address Translation) gateway, many legitimate users might appear to share the same IP address, leading to unfair blocking.

Step-by-Step: Configuring Rate Limits in the Azure Portal

1. Navigate to your APIM instance: Open the Azure portal and locate your API Management service.
2. Select the API: In the left-hand menu, click on "APIs" and select the specific API you want to protect.
3. Choose the Scope: You can apply policies to the "All APIs" level, a specific API, or an individual operation. Choosing the operation level allows for the most granular control.
4. Open the Policy Editor: Click on the "Design" tab and then click the code icon (</>) in the "Inbound processing" section.
5. Insert the Policy: Place your rate-limit-by-key policy inside the <inbound> tag.

<inbound>
    <base />
    <rate-limit-by-key calls="50" 
                       renewal-period="30" 
                       counter-key="@(context.Subscription.Id)" />
</inbound>

6. Save: Click "Save" to apply the changes. The policy takes effect immediately.

Advanced Traffic Control: Quotas and Concurrency

Beyond simple rate limiting, you may need to control traffic based on total volume or concurrent requests.

Implementing Usage Quotas

Quotas are essential for monetizing APIs or ensuring that heavy users don't exhaust your backend resources. The quota policy works similarly to rate limits but covers a longer duration.

<quota calls="5000" renewal-period="2592000" />

The renewal-period is defined in seconds. 2,592,000 seconds corresponds to 30 days. This policy ensures that a specific user or subscription cannot exceed 5,000 requests per month.

Managing Concurrency

Sometimes, the issue isn't the total number of requests, but the number of simultaneous requests hitting your backend. If your backend service takes 5 seconds to process a request, 100 concurrent requests might cause a memory overflow. You can use the limit-concurrency policy to manage this.

<limit-concurrency key="backend-service-limit" max-count="20" />

This policy ensures that at any given moment, no more than 20 requests are being processed by the backend for the defined key. Any request exceeding this count will wait until a slot opens up or fail if the timeout is reached.

Warning: Be careful with limit-concurrency. If you set the max-count too low, you will artificially create a bottleneck at the gateway level, leading to increased latency for your users while they wait for a slot to open.

Best Practices for API Traffic Management

Implementing rate limiting is not a "set it and forget it" task. It requires an understanding of your traffic patterns and the capabilities of your backend.

1. Tiered Rate Limiting

Not all users are equal. You should implement different rate limits based on user tiers. For example, a "Free" tier might have a limit of 100 requests per hour, while a "Premium" tier might have 10,000 requests per hour. You can achieve this in APIM by checking the user's group or subscription properties.

2. Communicate Limits Clearly

When a user hits a rate limit, the API should return a 429 Too Many Requests status code. It is best practice to include headers that inform the client when they can retry. APIM does this automatically, but you should ensure your client applications are designed to handle these responses gracefully using exponential backoff.

3. Monitoring and Alerting

You cannot manage what you do not measure. Use Azure Monitor to track the number of 429 status codes returned by your gateway. A sudden spike in 429 errors might indicate that your current limits are too restrictive, or it might indicate a distributed denial-of-service (DDoS) attempt.

4. Use Global Policies for Global Protections

Always apply a baseline rate-limit at the "All APIs" level. This acts as a safety net. Even if a specific API is under-configured, the global policy will prevent the entire system from being taken down by a single runaway process.

Callout: The Importance of Graceful Degradation When your API is under heavy load, it is better to return a 429 error to some users than to allow the entire service to crash. A 429 error tells the client "try again later," which allows your system to recover. A system crash, however, provides no feedback and leaves the client with no path forward.

Common Pitfalls and How to Avoid Them

Pitfall 1: Overly Restrictive Limits

Setting limits that are too tight is the most common mistake. This leads to false positives where legitimate users are blocked.

• The Fix: Start by monitoring your traffic for a few weeks without any limits. Calculate the 95th percentile of request volume per user, and set your initial limit slightly above that.

Pitfall 2: Ignoring Distributed Environments

If your APIM instance is scaled out across multiple regions or nodes, the counter for your rate limit might be local to that node.

• The Fix: APIM handles distributed counters automatically for most scenarios, but if you are using custom logic, ensure you understand whether your counters are synchronized or local to the gateway instance.

Pitfall 3: Not Handling Retries in Client Applications

If your server returns a 429, but your client application immediately retries the request, you are just contributing to the traffic flood.

• The Fix: Implement "exponential backoff" in your client code. If a request fails with a 429, wait 1 second, then 2, then 4, and so on. This gives the server breathing room.

Comparison Table: Traffic Control Policies

Practical Example: Implementing a Subscription-Based Rate Limit

Imagine you have a scenario where you want to allow different request volumes based on the user's subscription type. We can use policy expressions to achieve this.

<inbound>
    <base />
    <choose>
        <when condition="@(context.Subscription.Name == 'Gold')">
            <rate-limit-by-key calls="1000" renewal-period="60" counter-key="@(context.Subscription.Id)" />
        </when>
        <when condition="@(context.Subscription.Name == 'Silver')">
            <rate-limit-by-key calls="500" renewal-period="60" counter-key="@(context.Subscription.Id)" />
        </when>
        <otherwise>
            <rate-limit-by-key calls="100" renewal-period="60" counter-key="@(context.Subscription.Id)" />
        </otherwise>
    </choose>
</inbound>

In this example, the policy checks the subscription name associated with the incoming request. If the user is "Gold," they get 1,000 requests per minute. If they are "Silver," they get 500. Everyone else is restricted to 100. This is a highly efficient way to manage service levels without changing the underlying backend code.

The Role of counter-key

The counter-key is the most important part of rate-limit-by-key. If you choose a key that is too broad (like context.Request.IpAddress behind a proxy), you might accidentally block legitimate users. If you choose a key that is too narrow (like context.Request.Url), you might not be effectively limiting the user's total impact. Always choose a key that uniquely identifies the entity you want to limit, such as context.Subscription.Id or a custom Header value.

Monitoring and Troubleshooting

When you implement these policies, you must have visibility into how they are performing. Azure APIM logs provide detailed information about why a request was blocked.

1. Enable Diagnostics: Ensure that your APIM instance is sending logs to an Azure Log Analytics workspace.
2. Querying 429s: Use Kusto Query Language (KQL) to find instances of throttled requests.

   ApiManagementGatewayLogs
   | where ResponseCode == 429
   | summarize count() by bin(TimeGenerated, 1h), OperationId
   

3. Analyze the results: If you see a high number of 429s for a specific operation, investigate who is making those requests. You may find that a specific client ID is consistently hitting the limit, indicating either a need for a higher tier or a bug in their application logic.

Integrating with External Systems

While APIM is great for local traffic control, sometimes you need to integrate with external systems for advanced throttling, such as Redis or a custom database. You can use the send-request policy to call an external service to check if a request should be allowed.

<inbound>
    <send-request mode="new" response-variable-name="auth-response">
        <url>https://my-auth-service.com/check-limit</url>
        <method>POST</method>
        <set-header name="Authorization" exists-action="override">
            <value>@(context.Request.Headers.GetValueOrDefault("Authorization"))</value>
        </set-header>
    </send-request>
    <choose>
        <when condition="@(((IResponse)context.Variables["auth-response"]).StatusCode != 200)">
            <return-response>
                <set-status code="429" reason="Too Many Requests" />
            </return-response>
        </when>
    </choose>
</inbound>

This approach allows you to move your complex business logic for throttling out of the APIM policy and into a dedicated microservice. This is useful for large enterprises with centralized traffic management systems.

Security Considerations: Throttling as a Defense

Throttling is a vital component of your security strategy. It is not just about performance; it is about availability.

DDoS Mitigation

While Azure provides specialized services like Azure DDoS Protection, APIM acts as a secondary line of defense. By setting strict rate-limit policies, you ensure that even if an attacker manages to bypass other layers, their ability to impact the backend is severely limited.

Brute Force Protection

If you have an authentication endpoint, you should apply a very strict rate-limit to it. An attacker trying to guess passwords will need to make thousands of requests. By limiting that endpoint to, for example, 5 requests per minute per IP address, you make brute-force attacks impractical.

<!-- Example: Protect Login Endpoint -->
<policies>
    <inbound>
        <choose>
            <when condition="@(context.Request.Url.Path.Contains("/login"))">
                <rate-limit-by-key calls="5" renewal-period="60" counter-key="@(context.Request.IpAddress)" />
            </when>
        </choose>
    </inbound>
</policies>

Summary of Key Takeaways

1. Proactive vs. Reactive: Use rate limiting to proactively manage traffic and quotas to manage usage tiers, while using concurrency limits to protect backend resources.
2. Granularity Matters: Always choose an appropriate counter-key. Using context.Subscription.Id is generally safer and more accurate than context.Request.IpAddress.
3. Communication is Vital: Ensure your APIs return clear status codes (429) and that your client applications implement exponential backoff to handle these responses correctly.
4. Monitor and Iterate: Use Azure Monitor and Log Analytics to observe how your rate limits are functioning. Adjust your limits based on real-world data, not just theoretical assumptions.
5. Security Integration: Use throttling not just for performance, but as a security mechanism to mitigate brute-force attacks on sensitive endpoints like login and password reset pages.
6. Tiered Strategy: Implement different limits for different types of users or services to ensure that your most important traffic is always prioritized.
7. Safety First: Always include a global "All APIs" rate limit to act as a final safety net against unexpected traffic surges or configuration errors.

By mastering these techniques, you transform your API gateway from a simple pass-through service into a robust traffic management engine that protects your infrastructure, enforces business rules, and provides a predictable experience for your users. Start small, monitor your results, and gradually refine your policies to match the unique needs of your ecosystem.