Custom Metrics and Telemetry

Mastering Custom Metrics and Telemetry in Azure Application Insights

Introduction: Why Custom Telemetry Matters

In the world of cloud-native development, standard logs and basic performance counters often provide only a surface-level view of application health. While Azure Application Insights automatically captures request rates, response times, and dependency failures, these "out-of-the-box" metrics rarely tell the whole story. Your business logic, unique user flows, and specific domain events are hidden from the platform unless you explicitly instrument your code to surface them. This process of sending custom telemetry is what transforms a generic monitoring dashboard into a powerful business intelligence tool.

By implementing custom metrics and telemetry, you bridge the gap between technical infrastructure monitoring and business performance analysis. Imagine you are running an e-commerce platform. Knowing that your API responds in 200ms is helpful, but knowing that the "Add to Cart" button is failing for 5% of users due to a specific inventory validation error is critical. Custom telemetry allows you to track these business-specific outcomes, correlate them with infrastructure performance, and proactively address issues before they impact your bottom line. This lesson will guide you through the architecture, implementation, and best practices of custom instrumentation, ensuring you have full visibility into your applications.

Understanding the Telemetry Types

Before diving into the implementation, it is vital to understand the four primary pillars of Application Insights telemetry. Each type serves a distinct purpose and is stored in specific schemas within the underlying Log Analytics workspace.

• Custom Metrics: These are numerical values that represent a point in time or an aggregate. They are ideal for tracking business KPIs like "Number of items in shopping cart," "Total value of processed orders," or "Temperature sensor readings." They are designed for fast aggregation and querying.
• Custom Events: These represent specific occurrences within your application. Use these for tracking user interactions, such as "User clicked login," "Document exported," or "Trial period started." Events allow you to count occurrences and analyze user behavior patterns.
• Custom Dependencies: While Application Insights captures standard SQL or HTTP calls automatically, you might need to track calls to internal legacy systems, custom message queues, or specialized hardware. You can manually record these to maintain a complete dependency map.
• Custom Traces: These are essentially structured log messages. You can send strings with varying severity levels (Information, Warning, Error) to record diagnostic information that doesn't fit into the metrics or events categories.

Callout: Metrics vs. Events It is a common point of confusion to distinguish between events and metrics. Think of an event as a "what happened" record—it captures a discrete action at a specific time. Think of a metric as a "how much" record—it represents a measurement that can be averaged, summed, or analyzed over a time range. If you want to know how many people clicked a button, use an event. If you want to know the average latency of a specific calculation, use a metric.

Setting Up the SDK

To start sending custom telemetry, you must first ensure your application is connected to an Application Insights resource. In most .NET, Java, or Node.js environments, this involves installing the appropriate SDK package and configuring the connection string.

Step-by-Step Configuration (C# Example)

1. Install the NuGet Package: Use the .NET CLI to add the telemetry client package to your project. dotnet add package Microsoft.ApplicationInsights
2. Initialize the Telemetry Client: In your dependency injection container (typically Program.cs or Startup.cs), register the client.
3. Inject and Use: Inject the TelemetryClient into your services or controllers to begin sending data.

// Example of service registration in .NET 6+
builder.Services.AddApplicationInsightsTelemetry();

// Example of usage in a controller
public class OrderController : ControllerBase
{
    private readonly TelemetryClient _telemetryClient;

    public OrderController(TelemetryClient telemetryClient)
    {
        _telemetryClient = telemetryClient;
    }

    [HttpPost]
    public IActionResult CreateOrder(Order order)
    {
        // Your logic here...
        _telemetryClient.TrackEvent("OrderCreated", new Dictionary<string, string> {
            { "OrderId", order.Id },
            { "TotalAmount", order.Amount.ToString() }
        });
        return Ok();
    }
}

Note: Always ensure that your TelemetryClient is registered as a singleton. Creating a new instance for every request is an anti-pattern that can lead to significant memory overhead and data loss due to the SDK's internal buffering mechanisms.

Implementing Custom Metrics

Custom metrics are powerful because they allow for granular monitoring of application state. In Application Insights, metrics are often sent as "MetricTelemetry" objects. However, for high-frequency data, it is more efficient to use the GetMetric API, which performs local aggregation before sending the data to the Azure backend.

Practical Example: Tracking Inventory Levels

Suppose you want to monitor the stock level of a product every time an order is processed. Instead of sending a telemetry item for every single change (which could be thousands per second), you should aggregate these locally.

// Define the metric once as a class member or static field
private static readonly Metric _stockLevelMetric = 
    telemetryClient.GetMetric("InventoryStockLevel", "ProductId");

// Update the metric
_stockLevelMetric.TrackValue(currentStockCount, productId);

The GetMetric approach is highly recommended because it reduces the number of messages sent over the network, lowering costs and reducing the performance impact on your application.

Comparison of Telemetry Approaches

Advanced Telemetry: Custom Dependencies and Traces

Sometimes your application interacts with systems that the Application Insights SDK cannot automatically detect. This is common when communicating with internal message buses, proprietary hardware interfaces, or legacy mainframe systems.

Tracking Custom Dependencies

To track these, you manually instantiate a DependencyTelemetry object. This allows you to provide a name, target, and duration, which enables the Application Insights "Application Map" to correctly visualize the connection.

var dependency = new DependencyTelemetry("InternalMessageBus", "QueueName", "ProcessMessage", DateTimeOffset.UtcNow, TimeSpan.FromMilliseconds(45));
dependency.Success = true;
telemetryClient.TrackDependency(dependency);

By providing the Duration and Success properties, you ensure that these custom dependencies appear in the performance charts alongside your standard SQL and HTTP dependencies. This creates a unified view of your application's external dependencies.

Best Practices for Tracing

While it is tempting to dump every variable into a trace, this practice creates "log noise" that makes it nearly impossible to find actionable information. Follow these guidelines:

1. Use Severity Levels: Always categorize your traces as TraceTelemetry with an appropriate SeverityLevel (Information, Warning, Error, Critical).
2. Structured Logging: Instead of concatenating strings like $"User {userId} failed", use property bags. This allows you to query logs easily in Kusto Query Language (KQL).
3. Avoid PII: Never include Personally Identifiable Information (PII) such as email addresses, social security numbers, or passwords in your telemetry.

Step-by-Step: Troubleshooting and Debugging Telemetry

Even with perfect code, telemetry can sometimes go missing. If your custom metrics aren't appearing in the Azure Portal, follow this systematic troubleshooting approach.

1. Check the Local Telemetry Channel

The Application Insights SDK buffers telemetry in memory. If your application crashes or exits abruptly, the buffer might not flush. You can force a flush for debugging purposes.

// Force a flush to the backend
telemetryClient.Flush();

2. Verify the Instrumentation Key / Connection String

Ensure your environment variables are correctly set. In Azure App Service, check the APPLICATIONINSIGHTS_CONNECTION_STRING setting in the Configuration blade. If the connection string is missing or incorrect, the SDK will silently drop all telemetry.

3. Inspect the Traffic with Fiddler or Wireshark

If you suspect the network is blocking the telemetry, use a proxy tool to verify that the application is actually attempting to reach dc.services.visualstudio.com. If you see 403 or 401 errors, your firewall or network policy is blocking the outgoing telemetry traffic.

4. Enable Developer Mode

During local development, the SDK defaults to a "sampling" behavior that aggregates data to save bandwidth. You can disable this to see every single event as it happens.

// In your startup/initialization
TelemetryConfiguration.Active.TelemetryChannel.DeveloperMode = true;

Warning: Never enable DeveloperMode in a production environment. It bypasses sampling and buffering, which will significantly increase your Azure costs and potentially degrade the performance of your production application.

Common Pitfalls and How to Avoid Them

Pitfall 1: Over-instrumentation (Metric Explosion)

Developers often think "more data is better." However, creating thousands of unique metric names (e.g., using a unique order ID as a metric name) will cause a "metric explosion." This crashes dashboard rendering and makes querying impossible.

• Solution: Use dimension names (e.g., Region, Status, Category) rather than unique values in the metric name. Always ensure your dimensions have a finite set of possible values.

Pitfall 2: Ignoring Sampling

By default, Application Insights might sample your data to stay within budget. If you are tracking critical business transactions, you might lose data points.

• Solution: Configure your TelemetryProcessor to exclude critical event types from sampling. You can explicitly tell the SDK to keep 100% of "OrderProcessed" events while sampling 10% of "PageVisited" events.

Pitfall 3: Failing to Correlate

Custom telemetry is only useful if you can relate it to the request that triggered it. If you send a custom event without a TelemetryContext, it will be an "orphan" record.

• Solution: Ensure you are using the TelemetryClient within the scope of an existing request. The SDK automatically attaches the OperationId and ParentId to any telemetry generated within a request-handling thread, which links your custom events to the parent web request.

Industry Standards and Governance

When instrumenting a large-scale enterprise application, individual developers should not be "guessing" what to track. Establish a telemetry strategy that includes:

1. Naming Conventions: Standardize how metrics are named. For example, use service.domain.metric_name (e.g., billing.payments.success_rate).
2. Schema Documentation: Maintain a central repository that defines what each custom event tracks and what properties it contains.
3. Access Control: Use Azure Role-Based Access Control (RBAC) to ensure that only authorized personnel can view sensitive telemetry data.
4. Cost Management: Set up "Daily Caps" on your Application Insights resource to prevent unexpected billing spikes caused by runaway logging loops.

Callout: The Importance of Context Always remember that context is king. A metric without context (like an OrderId, UserId, or Region) is just a number. By adding custom properties (dimensions) to your telemetry, you allow your team to filter, slice, and dice the data. Instead of just seeing "100 errors," you can see "100 errors in the 'Checkout' service for users in the 'US-East' region."

Advanced Querying with KQL

Once your custom telemetry is flowing into the Log Analytics workspace, you need to know how to retrieve it. KQL (Kusto Query Language) is the primary tool for this.

Example: Finding the Average of a Custom Metric

If you sent a custom metric named ProcessingTime, you can calculate the average over the last hour:

customMetrics
| where name == "ProcessingTime"
| where timestamp > ago(1h)
| summarize avg(value) by bin(timestamp, 5m)
| render timechart

Example: Analyzing Custom Events

If you want to see which users are triggering a specific event most frequently:

customEvents
| where name == "UserLogin"
| summarize count() by tostring(customDimensions.UserId)
| sort by count_ desc

This ability to drill down into the data is why custom instrumentation is so valuable. It turns raw logs into a narrative about how your users are interacting with your system.

Summary and Key Takeaways

Mastering custom metrics and telemetry in Azure Application Insights is a journey from being reactive to being proactive. By carefully instrumenting your application, you gain the ability to visualize, monitor, and alert on the things that truly matter to your business.

Key Takeaways:

• Think Before You Track: Don't instrument everything. Focus on business outcomes and critical failure paths. Over-instrumentation leads to performance degradation and high costs.
• Use the Right Tool: Choose the right telemetry type (Metric, Event, Trace, or Dependency) for the job. Use metrics for numerical trends and events for discrete user actions.
• Always Add Dimensions: A raw number is rarely enough. Add custom dimensions (e.g., UserId, Region, Status) to your telemetry to make the data actionable through filtering.
• Leverage Local Aggregation: Use the GetMetric API to aggregate high-frequency data locally before sending it to the cloud. This reduces network chatter and improves performance.
• Maintain Consistency: Follow a naming convention across your team. Inconsistent naming (e.g., LoginEvent vs User_Login) makes querying and dashboarding significantly harder.
• Test Your Telemetry: Treat your telemetry code like production code. Write unit tests for your instrumentation logic to ensure that events are fired correctly under different conditions.
• Monitor Your Costs: Keep an eye on your telemetry volume. Use sampling for non-critical data and set up alerts for when your daily data ingestion exceeds expected thresholds.

By following these principles, you will ensure that your Azure-based solutions remain observable, maintainable, and aligned with your business goals. Remember that observability is not a one-time setup; it is a continuous process of refining your instrumentation as your application evolves and your business requirements shift. Start small, identify your most critical KPIs, and build your monitoring strategy from there.