Report metrics to Azure Monitor Application Insights

Introduction

Azure Application Insights (App Insights) provides monitoring and alerting capabilities that can be leveraged to monitor the health of NServiceBus endpoints. This sample demonstrates how to report metric data to Azure Application Insights and present it graphically:

The sample reports the following metrics to App Insights:

• Fetched messages per second
• Failed messages per second
• Successful messages per second

The sample also shows data like Average message processing time in milliseconds and Maximum critical time in milliseconds. For a detailed explanation of these metrics refer to the metrics captured section in the metrics documentation.

Prerequisites

To run this sample, an App Insights account and instrumentation key are required.

Code overview

The sample simulates message load with a random 10% failure rate using the LoadSimulator class:

var simulator = new LoadSimulator(endpointInstance, TimeSpan.Zero, TimeSpan.FromSeconds(10));
await simulator.Start()
    .ConfigureAwait(false);

Connect to Azure Application Insights

To connect the sample code to App Insights, the instrumentation key must be provided.

By default the sample code loads this key from an environment variable called ApplicationInsightKey. Either set this environment variable or paste the instrumentation key in the following section.

var telemetryConfiguration = new TelemetryConfiguration(instrumentationKey);

endpointConfiguration.EnableApplicationInsights(telemetryConfiguration);

The instrumentation key can be retrieved from the Azure Portal by locating the App Insights instance, and then navigating to the Properties view.

Use NServiceBus.Metrics to capture metric values

To capture the metric values generated by NServiceBus, a class called ApplicationInsightsFeature is added to the sample. It is responsible for enabling the NServiceBus metrics feature.

configuration.EnableMetrics();

ProbeCollector ensures that the telemetry probe is subscribed to the event stream. During this stage ProbeCollector is configured and wired. Information such as endpoint name and input queue are passed to allow advanced data analysis in App Insights.

var instance = Guid.NewGuid().ToString("N");

var endpointName = settings.EndpointName();

var telemetryConfiguration = settings.Get<TelemetryConfiguration>();

collector = new ProbeCollector(
    telemetryConfiguration,
    endpointName,
    instance,
    context.LocalQueueAddress(),
    context.InstanceSpecificQueueAddress()
);

var metrics = settings.Get<MetricsOptions>();

metrics.RegisterObservers(collector.RegisterProbes);

Events and metrics are buffered and the App Insights telemetry client does this every minute. If the endpoint instance would gracefully stop it must ensure that any pending events are flushed to App Insights.

class CleanupAtStop : FeatureStartupTask
{
    public CleanupAtStop(ApplicationInsightsFeature instance)
    {
        this.instance = instance;
    }

    protected override Task OnStart(IMessageSession session, CancellationToken cancellationToken = default)
    {
        return Task.CompletedTask;
    }

    protected override Task OnStop(IMessageSession session, CancellationToken cancellationToken = default)
    {
        instance.collector.Flush();
        return Task.CompletedTask;
    }

    ApplicationInsightsFeature instance;
}

Sending data to App Insights

The ProbeCollector is used to pass NServiceBus-captured metric data to the App Insights instance configured earlier. Information is added as telemetry context properties, which will be attached to every metric (NServiceBus signal probe) and event (NServiceBus duration probe) captured via the telemetry client. The telemetry client is used to communicate with App Insights.

telemetryClient = new TelemetryClient(telemetryConfiguration);
var properties = telemetryClient.Context.GlobalProperties;
properties.Add("Endpoint", endpointName);
properties.Add("EndpointInstance", instance);
properties.Add("MachineName", Environment.MachineName);
properties.Add("HostName", Dns.GetHostName());
properties.Add("EndpointQueue", queueAddress.ToString());

if (instanceAddress != null)
{
    properties.Add("EndpointDiscriminator", instanceAddress.Discriminator);
}

When the ProbeCollector is registered with the NServiceBus metrics API, it registers observer callbacks to receive probe names, signals and durations. Signals are converted into App Insights custom events. Durations are converted into App Insights metrics.

foreach (var duration in context.Durations)
{
    if (!probeNameToAiNameMap.TryGetValue(duration.Name, out var name))
    {
        continue;
    }
    duration.Register(
        observer: (ref DurationEvent @event) =>
        {
            var milliseconds = @event.Duration.TotalMilliseconds;
            var telemetry = new MetricTelemetry(name, milliseconds);
            telemetryClient.TrackMetric(telemetry);
        });
}

foreach (var signal in context.Signals)
{
    if (!probeNameToAiNameMap.TryGetValue(signal.Name, out var name))
    {
        continue;
    }
    signal.Register(
        observer: (ref SignalEvent @event) =>
        {
            telemetryClient.TrackEvent(new EventTelemetry(name));
        });
}

Creating charts

To create the charts, metrics provided by NServiceBus, as well as signals and durations must be converted into App Insights metrics and events.

One way to set up graphs for metrics in App Insights is to go to the Metrics Explorer, add a new graph selecting the custom metric of choice and then use the push pin symbol to add it to a dashboard.

The following graphs show the following metrics:

• Maximum critical time & critical time grouped by instance id.
• Average processing time, not grouped.
• Minimum hours countdown until Service Level Agreement violation.

Similar for events, the following graphs show all events grouped by event name. This graph can be further filtered, for example, to show failures only.

Refer to the Microsoft documentation for more information on creating dashboards from the metrics explorer.

Using the metrics explorer is an easy way to get started, but to create truly advanced graphs, App Insights offers an advanced analytics environment.