Azure Monitor Alerts Configuration

Lesson: Configuring Azure Monitor Alerts for Cosmos DB

Introduction: Why Monitoring Matters for Cosmos DB

Azure Cosmos DB is a globally distributed, multi-model database service designed for high availability and low latency. Because it is a managed service, you do not have to worry about patching servers or managing storage hardware. However, the responsibility for ensuring the health, performance, and cost-efficiency of your database remains firmly in your hands. Without a proactive strategy, performance degradation or unexpected spikes in resource consumption can go unnoticed until they impact your end-users.

Monitoring is the practice of observing the telemetry data generated by your database to understand its current state. Azure Monitor is the primary tool for this purpose, providing a unified platform to collect, analyze, and act on data from your Azure resources. Configuring alerts within Azure Monitor allows you to transition from a reactive posture—where you fix problems after users report them—to a proactive one, where you address issues before they cause downtime.

Effective alerting for Cosmos DB is not just about catching errors; it is about maintaining a balance between performance and cost. For example, if your application experiences a sudden burst in traffic, your Request Units (RUs) might be exhausted, leading to 429 "Too Many Requests" errors. If you have an alert configured for RU consumption, you can respond by scaling your throughput or investigating the query patterns that caused the spike. This lesson explores how to design, implement, and manage a strategy for Cosmos DB alerts that ensures reliability and operational excellence.

Understanding Cosmos DB Telemetry

Before you can configure meaningful alerts, you must understand what data is available. Azure Cosmos DB emits several categories of metrics that provide insight into the engine's behavior. These metrics are accessible through the Azure Portal, the Azure CLI, PowerShell, and the REST API.

Key Metrics to Monitor

• Total Requests: The total number of requests sent to the database. This helps you understand your traffic patterns and identify unexpected volume.
• Normalized RU Consumption: This is arguably the most important metric. It represents the maximum RU consumption across any physical partition in your database or container. It is a percentage-based metric that helps you identify bottlenecks.
• 429 Errors (Request Rate Too Large): These occur when your application exceeds the provisioned throughput. High counts of these errors directly correlate to a poor user experience.
• Data Usage: Monitoring the storage size of your database is essential for capacity planning and cost management.
• Latency (p99): The time taken for requests to complete. High latency often points to inefficient queries or regional network issues.

Callout: Metric Aggregation Explained When setting up alerts, you must choose an "Aggregation Type." For example, if you choose "Average," the system calculates the mean value over a time window. If you choose "Maximum," the system looks for the single highest value during that window. For critical alerts like 429 errors or RU consumption, "Maximum" is often safer because it captures transient spikes that "Average" might smooth out and hide.

Designing an Alerting Strategy

A common mistake in monitoring is "alert fatigue." If your team receives hundreds of notifications a day, they will inevitably start ignoring them. A successful alerting strategy focuses on the metrics that actually indicate a problem that requires human intervention.

Tiering Your Alerts

1. Critical (Action Required): These alerts indicate an immediate impact on users. Examples include high rates of 429 errors, service unavailability, or extreme latency spikes. These should trigger immediate notifications (SMS, PagerDuty, or email to on-call engineers).
2. Warning (Investigation Required): These alerts indicate that a threshold is being approached. For instance, if RU consumption hits 80%, you might want an alert so that a developer can check if a re-indexing job or a massive data import is running.
3. Informational (Trend Analysis): These are not for immediate action but for tracking. You might track storage growth trends over a month to prepare for budget meetings or to adjust your partitioning strategy.

Note: Always associate alerts with an Action Group. An Action Group is a collection of notification preferences (email, SMS, push notifications) and automated actions (Azure Functions, Webhooks, Logic Apps) that trigger when the alert criteria are met.

Step-by-Step: Configuring an Alert in the Azure Portal

Configuring an alert through the Azure Portal is the most straightforward method for most teams. Follow these steps to set up an alert for high RU consumption.

1. Navigate to your Cosmos DB account: Open the Azure Portal and find your specific Cosmos DB resource.
2. Access the Monitor tab: In the left-hand navigation pane, look for the "Monitoring" section and select "Alerts."
3. Create a new alert rule: Click on "+ Create" and select "Alert rule."
4. Define the Signal: Click "Select signal." In the search bar, type "Normalized RU Consumption." Select the metric from the list.
5. Configure Logic:
   • Threshold: Choose "Static."
   • Operator: "Greater than."
   • Threshold Value: Set this to 80 (representing 80% consumption).
   • Aggregation: "Maximum."
   • Evaluation Frequency: Set to 1 minute or 5 minutes depending on how quickly you need to react.
6. Action Groups: Select an existing Action Group or create a new one. This is where you define who gets notified when the alert triggers.
7. Details and Review: Give the alert a meaningful name, such as "CosmosDB-High-RU-Consumption-Warning." Assign it a severity level (e.g., Sev 2 for Warning). Review the settings and click "Create."

Using Infrastructure as Code (IaC)

Manually clicking through the portal is fine for a single database, but it is not sustainable for enterprise environments with dozens or hundreds of instances. Using Bicep or Terraform ensures that your monitoring configuration is consistent across environments.

Example: Bicep Configuration for a Cosmos DB Alert

The following Bicep snippet demonstrates how to define a metric alert for 429 errors.

resource cosmosDbAlert 'Microsoft.Insights/metricAlerts@2018-03-01' = {
  name: 'CosmosDB-TooManyRequests-Alert'
  location: 'global'
  properties: {
    severity: 1
    enabled: true
    scopes: [
      resourceId('Microsoft.DocumentDB/databaseAccounts', 'my-cosmos-account')
    ]
    evaluationFrequency: 'PT1M'
    windowSize: 'PT5M'
    criteria: {
      'odata.type': 'Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria'
      allOf: [
        {
          name: 'High429Errors'
          metricName: 'TotalRequests'
          metricNamespace: 'Microsoft.DocumentDB/databaseAccounts'
          operator: 'GreaterThan'
          threshold: 100
          timeAggregation: 'Total'
          criterionType: 'StaticThresholdCriterion'
        }
      ]
    }
    actions: [
      {
        actionGroupId: '/subscriptions/sub-id/resourceGroups/rg/providers/Microsoft.Insights/actionGroups/my-action-group'
      }
    ]
  }
}

Explanation:

• evaluationFrequency: This determines how often the alert rule evaluates the condition. PT1M means every minute.
• windowSize: This is the look-back period. PT5M means the alert looks at the data collected over the last 5 minutes.
• timeAggregation: For "TotalRequests," we use "Total" to sum up all requests in that 5-minute window.

Tip: When using IaC, always store your Action Group resource ID in a variable or a shared configuration file. This prevents hard-coding IDs and makes it easier to update your notification channels globally.

Best Practices for Alerting

1. Avoid Over-Alerting

The most common mistake is setting thresholds too low. If you set an alert for 50% RU consumption, you will likely be bombarded with alerts during normal business hours. Instead, set thresholds that indicate a genuine deviation from the norm. If you have "Autoscale" enabled on your Cosmos DB, you might only care about alerts when the database reaches the maximum provisioned throughput.

2. Contextualize Your Notifications

Generic alerts that say "Threshold breached" are useless. Use the "Custom JSON payload" feature in Action Groups to pass relevant information to your ticketing system (like ServiceNow or Jira). Include the resource name, the specific metric value, and a link to the Azure Portal diagnostic logs to help the responder triage the issue immediately.

3. Use Dynamic Thresholds

For metrics that vary heavily based on the time of day, static thresholds can be problematic. Azure Monitor offers "Dynamic Thresholds" which use machine learning to calculate the expected range of a metric based on historical patterns. This is excellent for traffic metrics that naturally spike during business hours and drop at night.

4. Test Your Alerts

Never assume your alerting is working until you have tested it. You can manually trigger a test by using a script to artificially increase the load on your database or by temporarily lowering the threshold to a value you know the current metric exceeds. Ensure the email or SMS arrives and that the tone/content is appropriate.

Callout: Static vs. Dynamic Thresholds Static thresholds are best for metrics that should never exceed a fixed limit (e.g., storage capacity). Dynamic thresholds are superior for metrics that fluctuate based on workload patterns (e.g., request count or latency), as they reduce false positives that occur during predictable high-traffic periods.

Common Pitfalls and Troubleshooting

The "Silent" Alert

A common issue is creating an alert rule but failing to associate it with an Action Group. The alert will fire in the background, showing up in the "Alerts" dashboard, but no one will receive a notification. Always check the "Actions" tab when creating or editing an alert rule to verify that a notification channel is selected.

The "Flapping" Alert

Flapping occurs when a metric hovers right at the threshold boundary. It causes the alert to trigger "on" and "off" repeatedly, resulting in a flood of emails. To avoid this, use the "Sensitivity" settings in dynamic thresholds or increase the "windowSize" to smooth out transient spikes.

Missing Permissions

Ensure the identity creating or managing the alerts has the "Monitoring Contributor" role. Without this, you may be able to view metrics but lack the permissions to create or modify alert rules.

Troubleshooting Checklist

If your alerts aren't triggering:

1. Check the Metric: Does the metric exist for that specific resource?
2. Check the Time Window: Is the evaluation frequency too long?
3. Check the Action Group: Is the email address correct? Is the SMS number verified?
4. Check the Scope: Are you applying the alert to the correct subscription, resource group, and resource?
5. Check for Suppression: Are there any maintenance windows or suppression rules active?

Comparing Alerting Options

Advanced Monitoring: Integrating Log Analytics

While metric alerts are great for simple thresholds, sometimes you need deeper insight. By enabling "Diagnostic Settings" on your Cosmos DB account, you can stream logs to a Log Analytics Workspace. Once the data is in Log Analytics, you can write Kusto Query Language (KQL) queries to trigger alerts based on complex patterns.

For example, you could write a query to alert you if a specific, expensive query is being executed more than 50 times in an hour.

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.DOCUMENTDB"
| where Category == "QueryRuntimeStatistics"
| where queryText_s contains "SELECT * FROM c"
| summarize count() by bin(TimeGenerated, 1h)
| where count_ > 50

This level of granularity is impossible with standard metric alerts. Integrating Log Analytics allows you to monitor the behavior of your application, not just the health of the infrastructure.

Managing Alert Life Cycles

Alerting is not a "set it and forget it" task. As your application evolves, your monitoring needs will change. You should schedule quarterly reviews of your alert rules. Ask yourself:

• Have we received any "false positive" alerts this quarter?
• Are there any recurring issues that we aren't currently alerting on?
• Do our current Action Groups reflect the current team structure?
• Are there any alerts that no one is paying attention to?

Removing stale alerts is just as important as creating new ones. A cluttered monitoring environment makes it harder for the team to spot the truly critical signals.

Key Takeaways

1. Proactive vs. Reactive: Monitoring is a core component of operational excellence; by setting up alerts, you shift from reacting to failures to preventing them.
2. Focus on Impactful Metrics: Prioritize alerts for metrics that directly affect the end-user, such as RU consumption and 429 error rates.
3. Use Infrastructure as Code: Automate the deployment of your alert rules using Bicep or Terraform to ensure consistency and repeatability across your environments.
4. Action Groups are Critical: An alert rule without an Action Group is just a log entry. Ensure that notification channels are correctly configured and tested.
5. Avoid Alert Fatigue: Set thresholds that matter. If you are receiving too many alerts, investigate the root cause of the noise and adjust your thresholds or use dynamic thresholds to reduce false positives.
6. Leverage Log Analytics for Depth: For complex troubleshooting, use diagnostic logs and KQL to create alerts based on specific query patterns and application behavior.
7. Continuous Improvement: Treat your monitoring configuration as a living part of your architecture; review and prune your alert rules regularly to maintain clarity and focus.

FAQ: Common Questions

Q: How quickly do metrics update in Azure Monitor? A: Most metrics for Cosmos DB are available within 1 to 3 minutes. However, depending on the aggregation method, it may take slightly longer for the values to propagate through the monitoring pipeline.

Q: Can I send alerts to Microsoft Teams or Slack? A: Yes. You can use a Webhook action in your Action Group to send alerts to a Teams or Slack channel. You may need to use an intermediate service like Azure Logic Apps to format the JSON payload correctly for these platforms.

Q: Does enabling monitoring affect my Cosmos DB performance? A: No. Azure Monitor and Diagnostic Settings run on a separate control plane. They do not consume your provisioned RU throughput or impact the latency of your database operations.

Q: What happens if my Action Group is disabled? A: If an Action Group is disabled, the alert will still fire and appear in the Azure Portal, but no external notifications will be sent. It is a good practice to verify the status of your Action Groups if you suspect you are not receiving alerts.

Q: Should I alert on every error? A: No. Alerting on every single error will lead to noise. Focus on error rates or percentages. For example, alerting when the error rate exceeds 1% of total requests is usually more meaningful than alerting on a single failed request.